Back to Details

mediapipe-face-mesh

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

MediaPipe FaceLandmarkerによる顔メッシュ・ブレンドシェイプ実装パターン

基于MediaPipe FaceLandmarker的人脸网格与BlendShape实现方案

概要

概要

FaceLandmarkerを使った顔メッシュ検出フロー：初期化→detectForVideo呼び出し→468ランドマーク取得→ARKit互換ブレンドシェイプ出力→DrawingUtilsによる描画。
基于FaceLandmarker的人脸网格检测流程：初始化→调用detectForVideo→获取468个关键点→输出ARKit兼容BlendShape→使用DrawingUtils进行绘制。

FaceLandmarker の初期化

FaceLandmarker 的初始化

typescript
import { FaceLandmarker, FilesetResolver } from "@mediapipe/tasks-vision";

const vision = await FilesetResolver.forVisionTasks(
  "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@latest/wasm"
);

const faceLandmarker = await FaceLandmarker.createFromOptions(vision, {
  baseOptions: {
    modelAssetPath:
      "https://storage.googleapis.com/mediapipe-models/face_landmarker/face_landmarker/float16/1/face_landmarker.task",
    delegate: "GPU",
  },
  runningMode: "VIDEO", // "IMAGE" or "VIDEO"
  numFaces: 1,
  outputFaceBlendshapes: true,           // ARKit互換ブレンドシェイプを出力
  outputFacialTransformationMatrixes: true, // 顔の変換行列を出力
});
typescript
import { FaceLandmarker, FilesetResolver } from "@mediapipe/tasks-vision";

const vision = await FilesetResolver.forVisionTasks(
  "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@latest/wasm"
);

const faceLandmarker = await FaceLandmarker.createFromOptions(vision, {
  baseOptions: {
    modelAssetPath:
      "https://storage.googleapis.com/mediapipe-models/face_landmarker/face_landmarker/float16/1/face_landmarker.task",
    delegate: "GPU",
  },
  runningMode: "VIDEO", // "IMAGE" or "VIDEO"
  numFaces: 1,
  outputFaceBlendshapes: true,           // 输出ARKit兼容BlendShape
  outputFacialTransformationMatrixes: true, // 输出人脸变换矩阵
});

検出結果のデータ構造

检测结果的数据结构

typescript
interface FaceLandmarkerResult {
  faceLandmarks: NormalizedLandmark[][];       // 顔ごとに468点（正規化座標0-1）
  faceBlendshapes?: Classifications[];          // ブレンドシェイプ（要outputFaceBlendshapes: true）
  facialTransformationMatrixes?: Matrix[];     // 顔の変換行列（要outputFacialTransformationMatrixes: true）
}

// NormalizedLandmark: { x: number, y: number, z: number }
// faceBlendshapes[0].categories: Category[]
// Category: { categoryName: string, score: number }  // score: 0.0〜1.0
typescript
interface FaceLandmarkerResult {
  faceLandmarks: NormalizedLandmark[][];       // 每张脸对应468个点（归一化坐标0-1）
  faceBlendshapes?: Classifications[];          // BlendShape（需开启outputFaceBlendshapes: true）
  facialTransformationMatrixes?: Matrix[];     // 人脸变换矩阵（需开启outputFacialTransformationMatrixes: true）
}

// NormalizedLandmarker: { x: number, y: number, z: number }
// faceBlendshapes[0].categories: Category[]
// Category: { categoryName: string, score: number }  // score范围：0.0~1.0

detectForVideo の呼び出しパターン

detectForVideo 的调用方式

typescript
// requestAnimationFrameループ内で呼び出す
const nowInMs = performance.now();
const result = faceLandmarker.detectForVideo(videoElement, nowInMs);
typescript
// 在requestAnimationFrame循环内调用
const nowInMs = performance.now();
const result = faceLandmarker.detectForVideo(videoElement, nowInMs);

顔メッシュの描画パターン

人脸网格的绘制方式

typescript
import { DrawingUtils } from "@mediapipe/tasks-vision";

const drawingUtils = new DrawingUtils(ctx);
for (const landmarks of result.faceLandmarks) {
  // テセレーション（フルメッシュ）— 薄い半透明
  drawingUtils.drawConnectors(landmarks, FaceLandmarker.FACE_LANDMARKS_TESSELATION, {
    color: "rgba(200,200,200,0.3)",
    lineWidth: 0.5,
  });
  // 目
  drawingUtils.drawConnectors(landmarks, FaceLandmarker.FACE_LANDMARKS_RIGHT_EYE, {
    color: "#30FF30",
  });
  drawingUtils.drawConnectors(landmarks, FaceLandmarker.FACE_LANDMARKS_LEFT_EYE, {
    color: "#30FF30",
  });
  // 眉毛
  drawingUtils.drawConnectors(landmarks, FaceLandmarker.FACE_LANDMARKS_RIGHT_EYEBROW, {
    color: "#FF3030",
  });
  drawingUtils.drawConnectors(landmarks, FaceLandmarker.FACE_LANDMARKS_LEFT_EYEBROW, {
    color: "#FF3030",
  });
  // 顔輪郭
  drawingUtils.drawConnectors(landmarks, FaceLandmarker.FACE_LANDMARKS_FACE_OVAL, {
    color: "#FFFFFF",
  });
  // 唇
  drawingUtils.drawConnectors(landmarks, FaceLandmarker.FACE_LANDMARKS_LIPS, {
    color: "#FF69B4",
  });
}
typescript
import { DrawingUtils } from "@mediapipe/tasks-vision";

const drawingUtils = new DrawingUtils(ctx);
for (const landmarks of result.faceLandmarks) {
  // 三角剖分（完整网格）— 浅灰色半透明
  drawingUtils.drawConnectors(landmarks, FaceLandmarker.FACE_LANDMARKS_TESSELATION, {
    color: "rgba(200,200,200,0.3)",
    lineWidth: 0.5,
  });
  // 右眼
  drawingUtils.drawConnectors(landmarks, FaceLandmarker.FACE_LANDMARKS_RIGHT_EYE, {
    color: "#30FF30",
  });
  // 左眼
  drawingUtils.drawConnectors(landmarks, FaceLandmarker.FACE_LANDMARKS_LEFT_EYE, {
    color: "#30FF30",
  });
  // 右眉毛
  drawingUtils.drawConnectors(landmarks, FaceLandmarker.FACE_LANDMARKS_RIGHT_EYEBROW, {
    color: "#FF3030",
  });
  // 左眉毛
  drawingUtils.drawConnectors(landmarks, FaceLandmarker.FACE_LANDMARKS_LEFT_EYEBROW, {
    color: "#FF3030",
  });
  // 人脸轮廓
  drawingUtils.drawConnectors(landmarks, FaceLandmarker.FACE_LANDMARKS_FACE_OVAL, {
    color: "#FFFFFF",
  });
  // 嘴唇
  drawingUtils.drawConnectors(landmarks, FaceLandmarker.FACE_LANDMARKS_LIPS, {
    color: "#FF69B4",
  });
}

ブレンドシェイプ（ARKit互換、最大52カテゴリ）

BlendShape（ARKit兼容，最多52个类别）

browDownLeft, browDownRight, browInnerUp, browOuterUpLeft, browOuterUpRight
eyeBlinkLeft, eyeBlinkRight, eyeLookDownLeft, eyeLookUpLeft
eyeSquintLeft, eyeSquintRight, eyeWideLeft, eyeWideRight
jawForward, jawLeft, jawOpen, jawRight
mouthClose, mouthFunnel, mouthLeft, mouthRight, mouthSmileLeft, mouthSmileRight
mouthPucker, mouthShrugLower, mouthShrugUpper
noseSneerLeft, noseSneerRight
cheekPuff, cheekSquintLeft, cheekSquintRight
browDownLeft, browDownRight, browInnerUp, browOuterUpLeft, browOuterUpRight
eyeBlinkLeft, eyeBlinkRight, eyeLookDownLeft, eyeLookUpLeft
eyeSquintLeft, eyeSquintRight, eyeWideLeft, eyeWideRight
jawForward, jawLeft, jawOpen, jawRight
mouthClose, mouthFunnel, mouthLeft, mouthRight, mouthSmileLeft, mouthSmileRight
mouthPucker, mouthShrugLower, mouthShrugUpper
noseSneerLeft, noseSneerRight
cheekPuff, cheekSquintLeft, cheekSquintRight

ブレンドシェイプのトップN表示

显示Top N个BlendShape

typescript
const blendshapes = result.faceBlendshapes?.[0]?.categories ?? [];
const topN = [...blendshapes]
  .sort((a, b) => b.score - a.score)
  .slice(0, 5);
// topN.map(bs => `${bs.categoryName}: ${(bs.score * 100).toFixed(1)}%`)
typescript
const blendshapes = result.faceBlendshapes?.[0]?.categories ?? [];
const topN = [...blendshapes]
  .sort((a, b) => b.score - a.score)
  .slice(0, 5);
// topN.map(bs => `${bs.categoryName}: ${(bs.score * 100).toFixed(1)}%`)

ブレンドシェイプを名前で参照

通过名称获取BlendShape

typescript
const blendshapes = result.faceBlendshapes?.[0]?.categories ?? [];
const blendshapeMap = Object.fromEntries(
  blendshapes.map((bs) => [bs.categoryName, bs.score])
);

// 目の開閉状態
const eyeBlinkLeft = blendshapeMap["eyeBlinkLeft"] ?? 0;   // 0=開, 1=閉
const jawOpen = blendshapeMap["jawOpen"] ?? 0;             // 0=閉, 1=開
const mouthSmileLeft = blendshapeMap["mouthSmileLeft"] ?? 0;
typescript
const blendshapes = result.faceBlendshapes?.[0]?.categories ?? [];
const blendshapeMap = Object.fromEntries(
  blendshapes.map((bs) => [bs.categoryName, bs.score])
);

// 眼睛开合状态
const eyeBlinkLeft = blendshapeMap["eyeBlinkLeft"] ?? 0;   // 0=睁开, 1=闭合
const jawOpen = blendshapeMap["jawOpen"] ?? 0;             // 0=闭合, 1=张开
const mouthSmileLeft = blendshapeMap["mouthSmileLeft"] ?? 0;

顔の変換行列（顔向き・位置）

人脸变换矩阵（人脸朝向与位置）

typescript
if (result.facialTransformationMatrixes?.[0]) {
  const matrix = result.facialTransformationMatrixes[0].data; // Float32Array, 4x4列優先
  // Three.jsのMatrix4に変換可能
  // new THREE.Matrix4().fromArray(matrix)
}
typescript
if (result.facialTransformationMatrixes?.[0]) {
  const matrix = result.facialTransformationMatrixes[0].data; // Float32Array，4x4列优先
  // 可转换为Three.js的Matrix4
  // new THREE.Matrix4().fromArray(matrix)
}

ユースケース例

应用场景示例

  • バーチャルアバター（表情のリアルタイム転送）
  • 表情認識（感情分析、ユーザーリサーチ）
  • ARフィルター（サングラス、マスク等の顔へのオーバーレイ）
  • 3DキャラクターアニメーションへのARKitブレンドシェイプ適用
  • アクセシビリティ（視線・口形状によるUI操作）
  • 虚拟形象（表情实时传递）
  • 表情识别（情感分析、用户调研）
  • AR滤镜（在人脸上叠加墨镜、口罩等元素）
  • 将ARKit BlendShape应用于3D角色动画
  • 无障碍交互（通过视线、嘴部形状操作UI）