Google Research has introduced PhotoScan, an experimental deep-learning system that analyzes body images captured with a smartphone. The technology estimates body-fat percentage and fat distribution, using the results to improve the prediction of insulin resistance.

In a clinical research setting, adding PhotoScan measurements produced insulin-resistance classification results close to those achieved with DXA body scans. PhotoScan is not currently a medical product, however, and cannot replace blood tests or a doctor’s assessment.

Why BMI does not show the full picture

Body Mass Index is calculated from a person’s height and weight. It is simple and widely available, but it cannot distinguish fat from muscle or show where fat is stored.

Two people with the same BMI may have very different body composition and metabolic risk. Fat stored around internal organs, for example, is generally more relevant to metabolic health than subcutaneous fat stored beneath the skin.

DXA scanning can provide a more detailed assessment of body composition. The technology requires specialized clinical equipment, making it unsuitable for convenient and frequent at-home screening.

How PhotoScan works

PhotoScan analyzes frontal and side images of the body. The research model also uses sex, height, weight and calculated BMI rather than relying on a photograph alone.

It estimates three measurements:

• total body-fat percentage;

• the ratio of fat around the trunk to fat around the hips and thighs;

• the ratio of visceral fat to subcutaneous fat.

Google initially trained the neural network using records from more than 35,000 UK Biobank participants. It was then fine-tuned using smartphone images and DXA measurements collected from a separate group of 677 adults.

What the study found

In an independent validation cohort of 132 people, PhotoScan estimated body-fat percentage with a mean absolute error of approximately 2.13 percentage points compared with DXA.

A baseline model using age, sex and BMI achieved an AUROC of 0.692 for classifying insulin resistance. Adding PhotoScan measurements increased the result to 0.760. A comparable model using clinical DXA data reached 0.773.

These numbers show that PhotoScan approached DXA performance for this particular insulin-resistance classification task. They do not mean that smartphone photos are as accurate as DXA for every body-composition measurement or medical purpose.

Is PhotoScan available to smartphone users?

No. Google describes PhotoScan as an investigational research framework and has not announced a public app or consumer feature.

The independent validation group was relatively small, and the images were captured under controlled study procedures. Larger and more diverse clinical studies would be needed before the technology could be considered for routine medical use.

Even if a similar tool becomes publicly available, it would most likely serve as an initial risk-screening method rather than a diagnostic test. Insulin resistance must be assessed by a healthcare professional using laboratory results, medical history and other relevant information.

Sources: Google Research and the PhotoScan research paper.