How to save in .svg files? about im2vec

[nopperl]

@DYF-AI it should work if you change color[k] to self.colors[k] on line 281.

[DYF-AI]

/Im2Vec/logs/VectorVAEnLayers/version_110/models/vector_vae_nlayers.py", line 281, in save
color = make_tensor(color[k])
UnboundLocalError: local variable 'color' referenced before assignment

[preddy5]

Hey @DYF-AI
I have few deadlines to work on this week so I'll look into the issue next week.
Meanwhile heres the code I used to generate the images for the paper.

https://gist.github.com/preddy5/29db261676188dd075b95ebf24b2cd19
use it like this
save_svg(f"{save_dir}/{name}.svg", self.imsize, self.imsize, shapes, shape_groups)

shapes and shape_groups are the variables in the raster function here https://github.com/preddy5/Im2Vec/blob/master/models/vector_vae.py#L266

Regards,
Pradyumna.

[DYF-AI]

@DYF-AI it should work if you change color[k] to self.colors[k] on line 281.

(1) I change expriment.py in line 252
if save_svg:
self.model.save(test_input, save_dir, name) and run CUDA_VISIBLE_DEVICES=1 python3 eval_local.py -c configs/emoji.yaml

/Im2Vec/logs/VectorVAEnLayers/version_110/models/vector_vae_nlayers.py", line 281, in save
color = make_tensor(self.colors[k])
IndexError: list index out of range

[kelisiya]

@DYF-AI it should work if you change color[k] to self.colors[k] on line 281.

(1) I change expriment.py in line 252
if save_svg:
self.model.save(test_input, save_dir, name) and run CUDA_VISIBLE_DEVICES=1 python3 eval_local.py -c configs/emoji.yaml

/Im2Vec/logs/VectorVAEnLayers/version_110/models/vector_vae_nlayers.py", line 281, in save
color = make_tensor(self.colors[k])
IndexError: list index out of range

Have you solved this problem and save .svg files ?

[kelisiya]

Hey @DYF-AI
I have few deadlines to work on this week so I'll look into the issue next week.
Meanwhile heres the code I used to generate the images for the paper.

https://gist.github.com/preddy5/29db261676188dd075b95ebf24b2cd19
use it like this
save_svg(f"{save_dir}/{name}.svg", self.imsize, self.imsize, shapes, shape_groups)

shapes and shape_groups are the variables in the raster function here https://github.com/preddy5/Im2Vec/blob/master/models/vector_vae.py#L266

Regards,
Pradyumna.

When I use this code to generator svg file ,
points = all_points[k].cpu().contiguous()#[self.sort_idx[k]] print(points.shape) print(points[4, 0])
IndexError: index 4 is out of bounds for dimension 0 with size 3
The input is input tensor ,can you provide a png to svg demo in your work, thanks ~

[preddy5]

Hey @kelisiya
Could you elaborate more on how you used the gist function I shared.

[kelisiya]

Hey @kelisiya
Could you elaborate more on how you used the gist function I shared.

First, I use the eval_local function to load the weights to predict the default data/train, and at the same time I put the save_svg function in vector_vae_nlayers.py to replace it. If I send the output result to save_svg, such an error will be reported. I guess it is because the output of the model is a comprehensive graph rather than a single result?

[preddy5]

@kelisiya sorry it looks like my suggestions were confusing, here https://gist.github.com/preddy5/a666d758674300366c280a2bbfe5c5ad this is the notebook I use for generating some results for the paper. Disclaimer: A lot of the code in there is not optimized or ready for release, but it should help you out.

[kelisiya]

@kelisiya sorry it looks like my suggestions were confusing, here https://gist.github.com/preddy5/a666d758674300366c280a2bbfe5c5ad this is the notebook I use for generating some results for the paper. Disclaimer: A lot of the code in there is not optimized or ready for release, but it should help you out.

thank you very much ~ I can run the demo , if I want to draw my svg , I should train the new model ? I selet some png to data file and run the inference , I found the result error . For example , I use airplane.png to data file , the result still is emoji_u1f632's result .

[athena913]

Using the save_svg code directly did not generate the desired output even after fixing the indexing errors mentioned above. I think the key is to figure out how to generate the points from the reconstructed image. Here's an alternative solution that I used.

The code in painterly_rendering.py at https://github.com/BachiLi/diffvg/blob/master/apps/painterly_rendering.py generates svg output from an image input. It uses backprop to iteratively generate an SVG output that matches the input. I used this with the code in sample_interpolate() in experiment.py to generate a SVG output for example, for the first reconstructed image (recons[0]) as follows:

 if save_svg:
             #self.model.save(test_input, save_dir, name) #original code
             genSVG(recons[0], save_dir)  

The following is the code for genSVG based on the code in painterly_rendering.py. The original code loads the input image from a file. I have modified it to take as input an image generated by the VectorVAEnLayers model (used in experiment.py).
This code generates an SVG output periodically during the optimization (to minimize the LPIPS loss) and you can specify the number of iterations (num_iter) to achieve the desired output.

import pydiffvg
import torch
import skimage
import skimage.io
import random
import ttools.modules
import math

pydiffvg.set_print_timing(True)

gamma = 1.0

def genSVG(inp_img, save_dir):
    
    num_paths = 512
    max_width = 2.0
    use_lpips_loss = True
    num_iter = 500
    use_blob = False
    
    # Use GPU if available
    pydiffvg.set_use_gpu(torch.cuda.is_available())
    
    perception_loss = ttools.modules.LPIPS().to(pydiffvg.get_device())
    
    #target = torch.from_numpy(skimage.io.imread('imgs/lena.png')).to(torch.float32) / 255.0
    #target = torch.from_numpy(inp_img).to(torch.float32) / 255.0
    target = inp_img.to(pydiffvg.get_device())
    target = target.pow(gamma)
    target = target.unsqueeze(0)
    print(target.shape, target)
    #target = target.permute(0, 3, 1, 2) # NHWC -> NCHW
    #target = torch.nn.functional.interpolate(target, size = [256, 256], mode = 'area')
    canvas_width, canvas_height = target.shape[3], target.shape[2]
    num_paths = num_paths
    max_width = max_width
    
    random.seed(1234)
    torch.manual_seed(1234)
    
    shapes = []
    shape_groups = []
    if use_blob:
        for i in range(num_paths):
            num_segments = random.randint(3, 5)
            num_control_points = torch.zeros(num_segments, dtype = torch.int32) + 2
            points = []
            p0 = (random.random(), random.random())
            points.append(p0)
            for j in range(num_segments):
                radius = 0.05
                p1 = (p0[0] + radius * (random.random() - 0.5), p0[1] + radius * (random.random() - 0.5))
                p2 = (p1[0] + radius * (random.random() - 0.5), p1[1] + radius * (random.random() - 0.5))
                p3 = (p2[0] + radius * (random.random() - 0.5), p2[1] + radius * (random.random() - 0.5))
                points.append(p1)
                points.append(p2)
                if j < num_segments - 1:
                    points.append(p3)
                    p0 = p3
            points = torch.tensor(points)
            points[:, 0] *= canvas_width
            points[:, 1] *= canvas_height
            path = pydiffvg.Path(num_control_points = num_control_points,
                                 points = points,
                                 stroke_width = torch.tensor(1.0),
                                 is_closed = True)
            shapes.append(path)
            path_group = pydiffvg.ShapeGroup(shape_ids = torch.tensor([len(shapes) - 1]),
                                             fill_color = torch.tensor([random.random(),
                                                                        random.random(),
                                                                        random.random(),
                                                                        random.random()]))
            shape_groups.append(path_group)
    else:
        for i in range(num_paths):
            num_segments = random.randint(1, 3)
            num_control_points = torch.zeros(num_segments, dtype = torch.int32) + 2
            points = []
            p0 = (random.random(), random.random())
            points.append(p0)
            for j in range(num_segments):
                radius = 0.05
                p1 = (p0[0] + radius * (random.random() - 0.5), p0[1] + radius * (random.random() - 0.5))
                p2 = (p1[0] + radius * (random.random() - 0.5), p1[1] + radius * (random.random() - 0.5))
                p3 = (p2[0] + radius * (random.random() - 0.5), p2[1] + radius * (random.random() - 0.5))
                points.append(p1)
                points.append(p2)
                points.append(p3)
                p0 = p3
            points = torch.tensor(points)
            points[:, 0] *= canvas_width
            points[:, 1] *= canvas_height
            #points = torch.rand(3 * num_segments + 1, 2) * min(canvas_width, canvas_height)
            path = pydiffvg.Path(num_control_points = num_control_points,
                                 points = points,
                                 stroke_width = torch.tensor(1.0),
                                 is_closed = False)
            shapes.append(path)
            path_group = pydiffvg.ShapeGroup(shape_ids = torch.tensor([len(shapes) - 1]),
                                             fill_color = None,
                                             stroke_color = torch.tensor([random.random(),
                                                                          random.random(),
                                                                          random.random(),
                                                                          random.random()]))
            shape_groups.append(path_group)
    
    scene_args = pydiffvg.RenderFunction.serialize_scene(\
        canvas_width, canvas_height, shapes, shape_groups)
    
    render = pydiffvg.RenderFunction.apply
    img = render(canvas_width, # width
                 canvas_height, # height
                 2,   # num_samples_x
                 2,   # num_samples_y
                 0,   # seed
                 None,
                 *scene_args)
    pydiffvg.imwrite(img.cpu(), save_dir+'init.png', gamma=gamma)

    points_vars = []
    stroke_width_vars = []
    color_vars = []
    for path in shapes:
        path.points.requires_grad = True
        points_vars.append(path.points)
    if not use_blob:
        for path in shapes:
            path.stroke_width.requires_grad = True
            stroke_width_vars.append(path.stroke_width)
    if use_blob:
        for group in shape_groups:
            group.fill_color.requires_grad = True
            color_vars.append(group.fill_color)
    else:
        for group in shape_groups:
            group.stroke_color.requires_grad = True
            color_vars.append(group.stroke_color)
    
    # Optimize
    points_optim = torch.optim.Adam(points_vars, lr=1.0)
    if len(stroke_width_vars) > 0:
        width_optim = torch.optim.Adam(stroke_width_vars, lr=0.1)
    color_optim = torch.optim.Adam(color_vars, lr=0.01)
    # Adam iterations.
    for t in range(num_iter):
        print('iteration:', t)
        points_optim.zero_grad()
        if len(stroke_width_vars) > 0:
            width_optim.zero_grad()
        color_optim.zero_grad()
        # Forward pass: render the image.
        scene_args = pydiffvg.RenderFunction.serialize_scene(\
            canvas_width, canvas_height, shapes, shape_groups)
        img = render(canvas_width, # width
                     canvas_height, # height
                     2,   # num_samples_x
                     2,   # num_samples_y
                     t,   # seed
                     None,
                     *scene_args)
        # Compose img with white background
        img = img[:, :, 3:4] * img[:, :, :3] + torch.ones(img.shape[0], img.shape[1], 3, device = pydiffvg.get_device()) * (1 - img[:, :, 3:4])
        # Save the intermediate render.
        pydiffvg.imwrite(img.cpu(), save_dir+'/iter_{}.png'.format(t), gamma=gamma)
        img = img[:, :, :3]
        # Convert img from HWC to NCHW
        img = img.unsqueeze(0)
        img = img.permute(0, 3, 1, 2) # NHWC -> NCHW
        if use_lpips_loss:
            loss = perception_loss(img, target) + (img.mean() - target.mean()).pow(2)
        else:
            loss = (img - target).pow(2).mean()
        print('render loss:', loss.item())
    
        # Backpropagate the gradients.
        loss.backward()

        # Take a gradient descent step.
        points_optim.step()
        if len(stroke_width_vars) > 0:
            width_optim.step()
        color_optim.step()
        if len(stroke_width_vars) > 0:
            for path in shapes:
                path.stroke_width.data.clamp_(1.0, max_width)
        if use_blob:
            for group in shape_groups:
                group.fill_color.data.clamp_(0.0, 1.0)
        else:
            for group in shape_groups:
                group.stroke_color.data.clamp_(0.0, 1.0)

        if t % 10 == 0 or t == num_iter - 1:
            pydiffvg.save_svg(save_dir+'iter_{}.svg'.format(t),
                              canvas_width, canvas_height, shapes, shape_groups)
    
    # Render the final result.
    img = render(canvas_width, #target.shape[1], # width
                 canvas_height, #target.shape[0], # height
                 2,   # num_samples_x
                 2,   # num_samples_y
                 0,   # seed
                 None,
                 *scene_args)
    # Save the intermediate render.
    pydiffvg.imwrite(img.cpu(), save_dir+'final.png'.format(t), gamma=gamma)
    # Convert the intermediate renderings to a video.
    from subprocess import call
    call(["ffmpeg", "-framerate", "24", "-i",
        save_dir+"iter_%d.png", "-vb", "20M",
        save_dir+"out.mp4"])





 

[preddy5]

I apologize I didnt realize that the python notebook URL is throwing an error, here's a better one https://gist.github.com/preddy5/68bf5cf78a147cb4ae58d6698d095f93
Hey @athena913 thank you for the suggestion, yes the code you share should give you a vector graphic output however I don't think that it is the most optimal solution since it would give you a vector graphic with 512 paths, instead if you save the vectors estimated by the im2vec decoder in the case of emojis the same image would be represented with 4paths.