Thank you for the thoughtful comment! Let me know if I miss any points:
1) We capture all attributes for elements and although we don't yet let you fail a test if the attributes are incorrect, that's where we're headed. So, you could imagine enabling/disabling accessibility validation at the account or per-test level. This is great feedback.
2) Re: pricing - thanks for this idea! Pricing is still evolving, will keep in mind.
3) We support parallel execution at no extra charge; it's only test execution time. So if you want to spin up 10 tests at once, that's fine. (We may need to scale up our infra at the outset, obviously.) There is no hard cap on the number of concurrent tests.
5) Runtime is the time the browser begins navigating to the page. You are not charged for time spent spinning up infrastructure or saving the test run. And just to be clear, you are not charged for any time spent recording tests.
I think it's all great news, and for customers it's reassuring to know that running time is really just about the test itself (which I noticed is in the FAQ, so sorry for wasting your time there). This makes me think about a couple more things: can clients select zones from which to run the tests? It's probably not a huge focus if your current target customers are in North America, but can become useful if a European company considers the tool and all loading times become noticeable. Or just if a company expects clients from anywhere and wants to test for that (which is basically the hardest bit for a small team, as it starts to involve having some devops experience to spin up VMs on the cloud in different zones and all that jazz).
Another tangentially related setting would be the locale for tests execution and scenarios (I'm thinking of date formats, monetary symbols, etc.. with expectations being different if you're in the US or in the UK for example)
The lack of cross-browser testing is something I can see scaring away some teams, especially with mobile being restricted to profiles of older iPhones only. IIRC over 70-80% of iPhone users are on the latest major iOS version. But it all ties in to other issues that I'm sure you're already dealing with (iPhone users using Safari, something like 20% of desktop users using Firefox/Safari, and the odd clients here and there who would have very specific annoying requirement like "I must be able to test on Internet Explorer - yes I know it's deprecated but my startup is B2B and I'm the small fish there").
Your docs might be short but they are already good and readable, especially for a target that might include a mix of pure devs and automation QAs/SDET/etc.
1) We capture all attributes for elements and although we don't yet let you fail a test if the attributes are incorrect, that's where we're headed. So, you could imagine enabling/disabling accessibility validation at the account or per-test level. This is great feedback.
2) Re: pricing - thanks for this idea! Pricing is still evolving, will keep in mind.
3) We support parallel execution at no extra charge; it's only test execution time. So if you want to spin up 10 tests at once, that's fine. (We may need to scale up our infra at the outset, obviously.) There is no hard cap on the number of concurrent tests.
4) Our API (https://reflect.run/docs/developer-api/documentation/) is in the early stages, but includes endpoints for executing tests and fetching results for tests in JSON.
5) Runtime is the time the browser begins navigating to the page. You are not charged for time spent spinning up infrastructure or saving the test run. And just to be clear, you are not charged for any time spent recording tests.
Thanks again!