roboticslab-uc3m / textiles

Research on algorithms for garment perception, manipulation...

textiles

---

Main repository for the research related to textile perception and manipulation at the University Carlos III of Madrid.

Textiles: Manipulation and Perception of Deformable Objects

The Textiles repository currently hosts two main lines of research:

Garment Unfolding

In [1] we developed a garment-agnostic algorithm to unfold clothes that works using 3D sensor information. The depth information provided by the sensor is converted into a grayscale image. This image is segmented using watershed algorithm. This algorithm provide us with labeled regions, each having a different height. In this labeled image, we assume that the highest height region belongs to the fold. Starting on this region, and ending in the garment border, tentative paths are created in several directions to analyze the height profile. For each profile, a bumpiness value is computed, and the lowest one is selected as the unfolding direction. A final extension on this line is performed to create a pick point on the fold border, and a place point outside the garment. In [2] we offer an extended description of the same original algorithm.

Later on, we wrote about future trends we expect in this field [3].

We have been developing this algorithm and extend its application to other kind of robots, such as industrial manipulators. A novel enhanced version of the algorithm with extended experimental validation has been sent and is currently under review.

Garment Ironing

We have developed an algorithm that allows a humanoid robot to iron garments with unmodified human tools. This work has been submitted for peer-reviewed publication and it is currently under review.

Repository

Structure of the repository:

• /images
• /ironing - robotic garment ironing applications developed with TEO the humanoid robot
• /unfolding - robotic garment unfolding applications developed with TEO the humanoid robot.
• /unfolding-industrial - robotic garment unfolding applications developed with industrial robots. (Under development).

Publications

Here you can find listed all publications related to the code hosted in this repository.

[1] David Estevez, Juan G. Victores, Santiago Morante, Carlos Balaguer. Towards Robotic Garment Folding: A Vision Approach for Fold Detection. International Conference on Autonomous Robot Systems and Competitions (ICARSC). 2016. [PDF] [poster] [bib] [URL]

[2] David Estevez. Towards Robotic Clothes Folding: A Garment-Agnostic Unfolding Algorithm. Master's Thesis. 2016. [PDF] [slides] [bib] [URL]

[3] David Estevez, Juan G. Victores, Carlos Balaguer. Future Trends in Perception and Manipulation for Unfolding and Folding Garments. Open Conference on Future Trends in Robotics (RoboCity 2016). 2016. [PDF] [slides] [bib] [URL]

textiles's Issues

Add Jupyter notebook as doc for Thesis Unfolding Algorithm

Document the Master's Thesis Unfolding Algorithm with a Jupyter notebook

Refactor generate_data.py

• Rename -> process kinfu
• Option for single folder
• Option for not recomputing already computed meshes
• Move to apps
• Auto-detect textures
• Fix output errors

More methods for comparison (ironing) required

Add more methods for comparison in the ironing algorithm.

Unmerged 'mover' branch

'Mover' branch is unmerged, and I'm not sure that it will be merged right away.

Could you merge it to develop @jgvictores ?

Check normalization of WiLD descriptor image

Check normalization of WiLD descriptor image, as it may affect results.

KinectFusion version without GUI

Currently KinectFusion (PCL implementation) requires using a graphical session to display its OpenGL-based GUI. This includes not only the data adquisition step, but also the texture baking step. It also requires input from the user to finish (pressing the 'S' key to save the data), so it cannot be launched and controlled from other program automatically (unless keystrokes are emulated).

There are some applications that would benefit from a headless KinectFusion that could be integrated in the software pipeline and automated, as well as run in a headless server.

Depth segmentation for removing the table is not enough

After fitting the table plane with RANSAC, some points of the table still remain in the point cloud.

Usign color information may improve garment segmentation (e.g. to apply grabcut)

Edit: this refers to 3D approach

Document Requirements for Ironing software

Ironing (frangi filter) requires scikit-image >= 0.13, which currently is a development version. To install it, we need to follow these steps:

1. Clone development repository:

$ git clone https://github.com/scikit-image/scikit-image.git

2. Upgrade Cython to version cython>=0.21 (if that version isn't already installed)

$ sudo -H pip3 install --upgrade cython

3. Installation:

$ cd scikit-image
$ sudo -H pip3 install -e .

Make old unfolding approach compatible with KinFu

Instead of a single frame + RGB approach, obtain data from a KinFu scan for the industrial unfolding application.

Unified installation Python-C++ through CMake and setuptools

Now that a textiles Python package exists, it's finally time to setup the installation process for this repo from CMake.

For this purpose, the following comment from @jgvictores in #25 will be very useful:

Not sure if relevant, but looks like @PeterBowman did a really good job at the asibot-main repository (see this, this and this).

Rename tool-asus-2.fcstd

Rename tool-asus-2.fcstd to tool-asus.fcstd (as we do not have a tool-asus)

Li's algorithm requires testing with actual garment data

Most of the implementation is currently done, but Li's algorithm requires testing with actual garment data.

Cluster selection less random

After clustering, the clusters are written to files. The naming of those clusters (i.e. cluster0 and cluster1) is totally random, as it depends on the choice of centroids by the k-means algorithm. This causes repeatability issues in the algorithm, as sometimes the ironing board is selected as garment.

Python wrappers for PCL-based algorithms

Currently the PCL-based code for algorithms is compiled and then executed from Python, using files to exchange data from PCL (C++) code to Python code.

It would be more convenient to have this code compiled as a Python module that could be used directly in the Python modules.

Refactor old code with new Transform class

Old code didn't use it (because it didn't exist).

Convert data from depth map to point cloud

Convert data from depth map to point cloud

Global metric of ironing success required

Implement a global metric of ironing success (as described in the paper)

Refactor ImageCreator clases

Refactor ImageCreator clases and unify interface:

• ImageCreator interface
• Old ImageCreator to use new interface
• New ImageCreators from unfolding-industrial

Ironing software usage documentation required

Ironing software usage documentation required

Evaluate performance with GPU PCL functions

Evaluate performance with GPU PCL functions to check if running time is reduced.

Refactoring broke "Get valid paths"

Valid paths filters all the paths

Refactoring according to Thesis

Refactor code to follow the Thesis structure. Then a Jupyter Notebook can be created as documentation.

ToDo list:

• Refactor "Preprocess"
• Refactor "Superpixels"
• Refactor "ClothContour"
• Refactor "intersect"

Review clustering algorithm

Clustering algorithm does weird things for some garments. Check out to discard bugs and try other potential approaches.

"textilesMover1" not a very descriptive name

I would suggest something like "TEOinterface" or "RobotInterface"

Or "RoboFolderInterface"

OpenCV3 breaks code

As they have changed the API of common methods such as findContours or imread.

Potentially affected files:

• ./ironing/perception/Li/DiscontinuityScanLi.py
• ./ironing/perception/ironingPathPlanning.py
• ./unfolding_industrial/perception/GarmentMirrorPickAndPlacePoints.py
• ./unfolding_industrial/perception/GarmentDepthSegmentation.py
• ./unfolding_industrial/apps/movement-test.py
• ./unfolding_industrial/apps/industrial-unfolding.py
• ./common/perception/Utils.py
• ./unfolding/perception/GarmentPickAndPlacePoints.py
• ./unfolding/perception/GarmentSegmentation.py
• ./unfolding/doc/Algorithm.ipynb
• ./unfolding/doc/.ipynb_checkpoints/Algorithm-checkpoint.ipynb
• ./unfolding/apps/textiles-evaluation.py
• ./unfolding/apps/main.py

Debug object does not close old windows

Caused by PointCloudLibrary/pcl#172

Import modules from textiles package does not work without Pycharm

When importing modules from the textiles package (e.g. common, ironing, etc) outside PyCharm an import error is raised.

This might be due to Pycharm including textiles submodules in the Python path, but needs to be fixed so that it can be run also outside PyCharm.

Color-based fiducial markers for calibration in point clouds

Support using color-based fiducial markers for calibration in point clouds (eye-hand transform).

RGB image <-> Kinfu Registration

A RGB image <-> Kinfu Registration is required to map pixels from the Kinect RGB capture to a ROI of the projected KinFu mesh.

This projected ROI would then be the input of the current CurvatureScan.

Improve README.md

Improve README.md in description, links, contributions...

Volume computation has to be normalized over each region

Currently, volume computation is computed in a naive way, using raw depth values. In order to avoid the effect of the underlying surface on the different volumes computed, each region should be normalized before computing the region volume.

Add depth adquisition to kinectGrabber

Currently kinectGrabber only saves an RGB frame.

Add depth adquisition to kinectGrabber to be able to save also the depth data. Ideally, ppm format can be used, as it allows 16bit resolution, but need to check this out.

Document dependencies

The whole project is missing dependencies (see #24 or #37 for instance). There are two main kinds of dependency here, Python-related and C++-related.

The main C++-related dependencies are:

• PCL
• Eigen
• OpenCV
• Freenect (for the KinectGrabber)
• yarp
• OpenMP (optional, for parallel calculations in ImageCreators)

The main Python-related dependencies are:

• Numpy
• matplotlib
• OpenCV 3 (this requires OpenCV contrib and a very specific version to use SVM loading functions for Li's algorithm)
• Scikit-image (requires development version as explained in #37)
• Scikit-learn (for clustering in some apps)
• yarp
• open_abb (from our open_abb fork, used in industrial-manipulation apps)
• pypcd (working with PCL point cloud files in Python, at ironing/perception/wrinkle_detection.pywhich might be a proof of concept code)
• tqdm
• StringIO and tabulate (at unfolding/perception/textiles-evaluation-stats.py, for generating tables from evaluation results)
• begins

Since #18 , maybe the best thing to do is document the requirements per module.

Better point selection criteria

Current ironing start/end points selection is good for simple skeletons, but complex shapes have less than ideal performance.

Refactor get_garment_contour to make it more Pythonic

get_garment_contour() has a dirty for loop that could be easily implemented using the max() method from standard Python.

Mirror points instead of extension

When doing the last step of our unfolding algorithm, the place point is obtained by extension of the pick point over the axis perpendicular to the fold.

It makes a lot more sense to use the fold as an axis to mirror the pick point to the place point.

Refactor Debug object to more descriptive name

Refactor Debug object to more descriptive name, as requested by @jgvictores

Possible choices are: DebugPointCloud, Debug3D, DebugViewer...

Points are transformed with z=0

When computing the pick and place points, points are transformed with z=0, so the computed points always lay on the table instead of the garment.

Allow to disable modules with CMake

Allow to disable modules with CMake to avoid compiling the whole project

Integrate old unfolding with robot platform for robotic unfolding

• Robot remote Cartesian movement
• Integration with old code (through new adapter)
• Robot remote digital output activation
• Data extraction (more or less automatic)
• Robot moving to desired points

Improve garment border detection

Garments on the ironing board are curved on the border due to the ironing board geometry.

Take this into account in the algorithm to discard false positives.

Path reconstruction does not deal with bifurcations

Path reconstruction does not deal with bifurcations, and sometimes weird things happen when reconstructing the trajectory (i.e. end point of trajectory is not the end point selected previously).

As a temporary solution I have added the following assertion to the code:

assert(path_points[-1][0]==end[0] and path_points[-1][1] == end[1])

to detect when the reconstruction fails, but this should be fixed.

Document DiscontinuityScanLi dependencies

Document how to install DiscontinuityScanLi dependencies:

• Begins:

  sudo pip3 install begins
  

• OpenCV 3.1 Python 3 bindings with SIFT features enabled (OpenCV_contrib)

  • Install OpenCV 3 with Python 3

pypcd does not work with Python 3

Functionality already requested in dimatura/pypcd#7

Complete vision pipeline in a single script

Refactor current vision pipeline for ironing to be launched from a single Python script for faster experiments.

Empty README.md

Write contents describing the project (and thesis, etc)

Experiments required for the whole ironing pipeline

Experiments required for the whole ironing pipeline

SVM training for wrinkle detection

This is the final step of Li's algorithm that we have to replicate.

Steps:

• Label data
• Extract SIFT
• Save SIFT to file
• Train SVM
• Fix loading SVM
• Get more data
• Prediction
• Test predictions
• Hough transform step to retrieve the wrinkle segments
• Non-maximum supression

Final step (bonus):

• Integration of Discontinuity and Curvature scans

Intention to move to Python 3

Python 3 has interesting and useful features and modules I'd like to use in this project, so I plan to update all existing code to use Python 3 instead of Python 2.

Improve garment segmentation in ironing

Current approach is effective, but requires fine-tunning of the segmentation parameters for each garment/color.

Possible ways of improving segmentation:

• Clustering