China’s Zhipu AI says its app can operate your smartphone for you

Zhipu AI is one of China's top start-ups working on generative AI. Photo: Handout

AutoGLM can order coffee, send messages and complete other tasks on smartphones based on voice prompts, according to the Chinese start-up

Chinese artificial intelligence (AI) start-up Zhipu AI is experimenting with a tool designed to complete tasks on behalf of smartphone users based on their voice commands, demonstrating a future where our digital lives are automated.

AutoGLM, an AI agent app, is able to understand relatively complex voice commands, such as "repeat my recent cereal order from shopping history", or "order a latte from the nearest cafe", according to the Beijing-based firm.

The tool can then plan out the steps involved in each task, "read" information that appears on screens, and perform the required actions on smartphones, said Zhipu AI, which runs a series of AI models and related chatbots, such as ChatGLM.

Do you have questions about the biggest topics and trends from around the world? Get the answers with SCMP Knowledge, our new platform of curated content with explainers, FAQs, analyses and infographics brought to you by our award-winning team.

AutoGLM is the latest example showing how Chinese start-ups are developing products to bring AI capabilities to consumers via smartphones, in a market where top foreign competitors are largely absent.

While Apple this week rolled out its on-device AI system, Apple Intelligence, in US English across most regions, the service is unavailable in mainland China, where the company is still working with local authorities to clear regulatory hurdles for its launch.

China is also excluded from the list of countries and regions where ChatGPT maker OpenAI and rival Anthropic offer access to their generative AI products. Last week, Anthropic debuted a feature similar to Zhipu AI's AutoGLM called "computer use", which automates certain computer operations, such as sorting out a spreadsheet or looking for specific information through thousands of rows of data.

As a product marketed towards Chinese users, AutoGLM is compatible with popular local apps.

When a Post reporter asked the AI agent to order a latte on Monday, for example, the app automatically opened food delivery platform Meituan, searched for cafes, ranked them by distance, chose the nearest one, and selected the correct drink. For safety reasons, the app still asked the user for manual confirmation before making a payment.

AutoGLM is currently able to operate with Tencent Holdings' multipurpose app WeChat, Alibaba Group Holding's Taobao marketplace, Alibaba-backed navigation app Amap, Meituan, crowdsourced review platform Dianping, and online travel agency Trip.com, among others. Alibaba owns the South China Morning Post.

Still, AutoGLM is far from perfect.

The AI agent remains in trial phase and is only available for download to invited Android users. It made several mistakes during a test by the Post. The AI agent sent WeChat messages to the wrong recipient, booked a hotel room for 30 days instead of the two days requested, and summarised a web page rather than writing a review on it when asked.

Zhipu AI noted in a user manual for early adopters that the tool's voice recognition module "still has lots of issues that need to be fixed".

Another common complaint from test users is delayed response.

The AI operates too slowly and still has much room to improve, according to one user in a WeChat group. The user added that unless one is too busy, it would be faster to complete the task yourself.