User:IIgwilo (WMF)/Real Time Preview for Wikitext

Test strategy for the Real Time Preview for Wikitext wish.

Project Documentation and High-level Requirements

• Original wish: Community_Wishlist_Survey_2021/Editing/Live_preview
• Project page: Real_Time_Preview_for_Wikitext

Quality Criteria

• The preview should be a faithful representation of how the article will look once published
• It should handle invalid wikitext in a way that helps the user recognise and correct errors
• Edits the user makes to the wikitext should be displayed in the preview in a reasonable amount of time
• Users should be happy with this feature from a usability perspective

Risks

• The real-time preview viewport is small. It might look different compared to viewing the article full-screen.
• Will a user be able to use this feature alongside other edit previewing features such as "Show changes"?

Oracles

• The real-time preview should match the "normal" preview
• The real-time preview should match the actual article once it is published

Coverage Criteria

• Coverage of wikitext (including invalid wikitext)
  • multimedia
  • templates
  • things which change geometry radically (e.g. tables, images)
  • things which change position on the screen (e.g. images when you float them)
  • Parser tests have lots of examples of wikitext we could use: gerrit:plugins/gitiles/mediawiki/core/+/refs/heads/master/tests/parser/ (every .txt file in that directory)
• Different content models, wikis, namespaces
• Section edits
• Browser coverage
• How this feature interacts with other features of the wikitext editor

Techniques

• HTML comparison
  • Compare the HTML of the preview with the HTML of the actual article (using a diff tool, for example)
• Stress test
  • What happens when we try to preview a huge amount of wikitext
    • E.g. try using preview on an article like en:Barack_Obama (no doubt there are even bigger articles)
  • How often can we update the preview in a short period of time? Can we overload it?

Test Environments

TBD. Probably beta primarily.