Updated the complete “Speak with Your Hands” tutorial to use the modern ml5.js HandPose API.Speak with your hands 2.0 by colemanliyah · Pull Request #1509 · processing/p5.js-website

colemanliyah · 2026-07-11T15:52:26Z

Resolves PR #1483 (#1483)

Replaced all older ml5 examples (ml5.handpose(), modelReady(), predict events, annotations) with the new API:
- await ml5.handPose()
- detectStart()
- keypoints
Updated every example sketch to work with the current ml5 version:
- Detecting hand keypoints
- Printing prediction data
- Drawing fingertips
- Adding images to fingertips
- Moving objects with fingers
- Controlling multiple objects with multiple fingers
- Using conditionals with finger positions
- Creating gesture-based interactions
Embedded all example sketches directly into the tutorial using instead of linking out to external sketches.
Added updated comments and explanations to match the new code structure.
Updated explanations of prediction data to match the new HandPose output format (keypoints, keypoints3D, handedness, confidence).

ksen0 · 2026-07-13T13:53:22Z

+    <meta charset="UTF-8" />
+    <title>Handpose with Webcam</title>
+
+    <script src="https://cdn.jsdelivr.net/npm/p5@2.0.5/lib/p5.js"></script>


Would it be possible to use p5@2 latest rather than a specific patch?

ksen0 · 2026-07-13T13:55:45Z

+createCanvas(400, 350);
+
+// Load Image
+img = await loadImage("https://raw.githubusercontent.com/ml5js/ml5-library/49b9bf2/examples/p5js/Handpose/Handpose_Image/data/hand.jpg");


Should this be a relative url to one of the assets you added?

ksen0

A fe minor comments, but thanks so much for going through and updating everything! @doradocodes if you verify locally as well please feel free to merge.

ksen0 · 2026-07-13T13:56:01Z

+createCanvas(400, 350);
+
+// Load the image and wait until it finishes loading
+img = await loadImage("https://raw.githubusercontent.com/ml5js/ml5-library/49b9bf2/examples/p5js/Handpose/Handpose_Image/data/hand.jpg");


Same here as above

shiffman

Hi @colemanliyah thanks for this wonderful work. I made some small comments throughout. Some of my comments apply to the subsequent examples too but I only noted it for the first instance of a change.

Also, the ml5.js style is to use a variable called hands rather than predictions. That might be a little friendlier to the reader, but it's also ok for this tutorial to have its own code style if you prefer to keep it as predictions.

Let me know if you have any questions!

shiffman · 2026-07-13T21:20:16Z

+async function setup() {
+// Create a canvas that's at least the size of the image.
+createCanvas(400, 350);


Small thing, but noting that the code isn't indented here.

shiffman · 2026-07-13T21:21:00Z

+// we use await to wait until the model is ready before continuing.
+handPose = await ml5.handPose();
+
+frameRate(1); // Set the frameRate to 1 since we don't need it to run quickly in this case.


Since this example incorporates noLoop() setting the frame rate to 1 seems extraneous to me.

shiffman · 2026-07-13T21:21:36Z

+handPose.detect(img, results => {
+predictions = results;
+});


For a single image this can actually be

Suggested change

handPose.detect(img, results => {

predictions = results;

});

predictions = await handPose.detect(img);

shiffman · 2026-07-13T21:22:13Z

+
+      fill(0, 255, 0);
+      noStroke();
+      ellipse(keypoint.x, keypoint.y, 10, 10);


circle() is a little friendlier!

Suggested change

ellipse(keypoint.x, keypoint.y, 10, 10);

circle(keypoint.x, keypoint.y, 10);

shiffman · 2026-07-13T21:25:39Z

+try {
+handPose = await ml5.handPose();
+console.log("HandPose loaded!");
+} catch (error) {
+console.error(error);
+}


I might skip the try catch here for simplicity.

shiffman · 2026-07-13T21:26:53Z

+  - `finger.points[3][0]` corresponds to the x-coordinate of the point 
+  - `finger.points[3][1]` corresponds to the y-coordinate of the point 


I think these should be .x and .y instead of [0] and [1]?

shiffman · 2026-07-13T21:29:02Z

+
+Let's change this code to add hats on each fingertip. [Full Sketch](https://editor.p5js.org/colemanliyah/sketches/pjzyE3gCp) Here are the steps:
+
+1. Add a variable to store the hat image and load it using the `preload()` function.


Change to loading with await!

shiffman · 2026-07-13T21:32:11Z

+- Replace the `drawKeypoints()` function with a new `drawObject()` function.
+- In `drawObject()`, access the index fingertip by selecting the last point of the index finger:
+  ```js
+  let tip = prediction.keypoints.slice(4, 8)[3];


I didn't comment on this for the previous examples b/c of the createFinger() method uses the arrays in an elegant way. However, here it would make a lot more sense to use the named points inside of the prediction object.

Suggested change

let tip = prediction.keypoints.slice(4, 8)[3];

let tip = prediction.index_finger_tip;

shiffman · 2026-07-13T21:33:49Z

+handPose.detectStart(video, results => {
+predictions = results;
+});


I would use a gotHands(results) callback here to match the ml5.js example style.

shiffman · 2026-07-13T21:34:22Z

+  let indexTip = prediction.keypoints.slice(4, 8)[3];
+  let middleTip = prediction.keypoints.slice(8, 12)[3];
+  let ringTip = prediction.keypoints.slice(12, 16)[3];


same here, use the named points directly.

colemanliyah added 3 commits July 10, 2026 15:24

Modifying speak with your hands examples

62f1390

Uploading hat image to create a link for the tutorial

03e0551

Modifying the rest of the examples to use modern version of ml5

1ca167f

colemanliyah mentioned this pull request Jul 11, 2026

[2.0] Fix "Speak with your hands" tutorial to work with p5 2.0 #1483

Open

ksen0 reviewed Jul 13, 2026

View reviewed changes

shiffman reviewed Jul 13, 2026

View reviewed changes

	ellipse(keypoint.x, keypoint.y, 10, 10);
	circle(keypoint.x, keypoint.y, 10);

		- `finger.points[3][0]` corresponds to the x-coordinate of the point
		- `finger.points[3][1]` corresponds to the y-coordinate of the point


		Let's change this code to add hats on each fingertip. [Full Sketch](https://editor.p5js.org/colemanliyah/sketches/pjzyE3gCp) Here are the steps:

		1. Add a variable to store the hat image and load it using the `preload()` function.

	let tip = prediction.keypoints.slice(4, 8)[3];
	let tip = prediction.index_finger_tip;

Uh oh!

Conversation

colemanliyah commented Jul 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ksen0 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shiffman left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

colemanliyah commented Jul 11, 2026 •

edited

Loading

shiffman left a comment •

edited

Loading