I'm impressed by the advances in Computer Use mentioned here and this got me wondering - is this already mature enough to be utilized for usability testing? Would I be right to assume that in general, a UI that is more difficult for AI to navigate is likely to also be relatively difficult for humans, and that it's a signal that it should be simplified/improved in some way?
Oh, I had assumed that it was trained on human interaction data and should be generally similar, and from the examples I saw - it generally was (although still not as good as us). In what sense do you expect it to be drastically different?
LLM engages with UI in a pragmatic way. It doesn't care about aesthetics or structural complexity, it doesn't care about latency or tactility. It doesnt have anywhere close to similar preferences and reasons for why it Vs a human being would and wouldn't click on or engage with something on a UI.
Do you really see no difference in the way a hollow algorithm interacts with UI vs how a person would go about interacting with UI?
