vulsio / go-cve-dictionary Goto Github PK

View Code? Open in Web Editor NEW

359.0 20.0 109.0 1016 KB

Build a local copy of CVE (NVD and Japanese JVN). Server mode for easy querying.

License: Apache License 2.0

Makefile 1.50% Go 96.50% Dockerfile 0.21% Python 1.79%

go-cve-dictionary's People

Contributors

Stargazers

Watchers

Forkers

fireowl11 netmarkjp xn0px90 sadayuki-matsuno ken5scal hiroakis noscripter mikecb yiyi99 pombredanne oswell ymomoi gleentea ykare yoheimuta gwiseman-r7 knqyf263 pyama86 galigalikun twiiiiin elfgoh valeriyaz ryumei chengfangang mai346 ngocnd0242 teitei-tk milk xmyl zhatin muitimon colorbox dmp42 tomdee theanalyz3r hibariya directionless zenithar primmus tobiasdai pioneerit godan jbmaillet takuzoo3868 amiangshu nicedabin scriptrock rockangator busterb dw-fans dookie18 sachinsangwanjr s-index segatomo santakd chroming famatte69 rich5 lubyruffy kazuminn mingunhuang noqcks hubspot chuanchang duckwed mrquiet samandersondon amarjitghuman nibiwodong shopper29 cybermonitor sbs2001 heizo yakshitbindal zwgirl medasz dogasantos mingwu123123 jhb92 yuanbing1113 mainek00n fdlucifer tttfrfr2 v1xingyue talhaorak gitwk86 i3ef0xh4ck ekmixon oppara sarthakadhikari isabella232 sthagen opoet7 compuwar y4ney aplunk infinipoint kannkyo sahar042 devinwangg

go-cve-dictionary's Issues

Change DB interface

https://github.com/kotakanbe/go-cve-dictionary/blob/master/db/db.go#L19-L20

In the current implementation, DB Error is only logging.
It's better to return an error.

fetchnvd does not retrieve CPE information

$ git rev-parse --short HEAD

bff11c4

$ make build

$ for i in `seq 2002 $(date +"%Y")`; do ./go-cve-dictionary fetchnvd -log-dir $(pwd)/log -years $i; done

CPE table is empty (and CPE lookup does not work).

Is there any other information you would need to help debug this?

Thanks.

dpath is not defined

go-cve-dictionary server -help
server:
        server
                [-bind=127.0.0.1]
                [-port=8000]
                [-dpath=$PWD/cve.sqlite3]
                [-debug]
                [-debug-sql]
.......

-dpath is not defined.
-dbpath is correct.

HTTP Proxy support

I am attempting to use this to fetch NVD's and am having issues with my HTTP proxy:

✔ ~/dev/vuls-data
12:52 $ go-cve-dictionary fetchnvd -years 2002
ERRO[0000] Failed to create log directory: mkdir /var/log/vuls: permission denied
 0 / 1 [--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------]   0.00%[Apr  3 12:53:03]  INFO Fetching... https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2002.xml.gz
[Apr  3 12:55:12] ERROR Failed to fetch cve data from NVD. err: [HTTP error. errs: [Get https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2002.xml.gz: dial tcp 129.6.13.177:443: getsockopt: connection timed out], url: https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2002.xml.gz]

Lock and manifest changed from json to toml

go version go1.8.1 linux/arm
error during make install
This seems to be the cause as dep now uses TOML

panic: interface conversion: interface {} is *toml.TomlTree, not []*toml.TomlTree

goroutine 1 [running]:
github.com/golang/dep/vendor/github.com/pelletier/go-toml.valueFromToml(0x1664200, 0x13e52a0, 0x1469700, 0xc420133900, 0xc420133900, 0x13e52a0, 0x13d03dd, 0x1d, 0x0)
	/Users/plgl757e/Projects/go/src/github.com/golang/dep/vendor/github.com/pelletier/go-toml/marshal.go:327 +0x223
github.com/golang/dep/vendor/github.com/pelletier/go-toml.valueFromTree(0x1664200, 0x1448480, 0xc4201338a0, 0x0, 0x0, 0x90, 0x0, 0x1657ea0)
	/Users/plgl757e/Projects/go/src/github.com/golang/dep/vendor/github.com/pelletier/go-toml/marshal.go:270 +0x2c6
github.com/golang/dep/vendor/github.com/pelletier/go-toml.Unmarshal(0xc420178000, 0x90, 0x600, 0x13d8760, 0xc42001b3e0, 0x0, 0xc42011b830)
	/Users/plgl757e/Projects/go/src/github.com/golang/dep/vendor/github.com/pelletier/go-toml/marshal.go:245 +0x1d6
github.com/golang/dep.readManifest(0x1658ca0, 0xc42000e140, 0xc42000e140, 0x0, 0x0)
	/Users/plgl757e/Projects/go/src/github.com/golang/dep/manifest.go:49 +0x16b
github.com/golang/dep.(*Ctx).LoadProject(0xc420130e70, 0x0, 0x0, 0x0, 0x0, 0x0)
	/Users/plgl757e/Projects/go/src/github.com/golang/dep/context.go:113 +0x415
main.(*ensureCommand).Run(0xc420133720, 0xc420130e70, 0xc42000c2e0, 0x0, 0x0, 0x0, 0x0)
	/Users/plgl757e/Projects/go/src/github.com/golang/dep/cmd/dep/ensure.go:110 +0x80
main.main()
	/Users/plgl757e/Projects/go/src/github.com/golang/dep/cmd/dep/main.go:125 +0x6df
make: *** [dep] Error 2

'make install' failed with 'dep ensure github.com/labstack/echo'.

I think that github.com/labstack/echo repository was changed.
so, I think it is necessary to execute 'dep ensure -add github.com/labstack/[email protected]' and update Gopkg.lock.
If I'm not wrong, I will push pull request.

$ make install
go get -u github.com/golang/dep/...
dep ensure -v
(1/35) Wrote gopkg.in/mattn/[email protected]
(2/35) Wrote github.com/hashicorp/go-version@master
(3/35) Wrote github.com/jinzhu/inflection@master
(4/35) Wrote github.com/asaskevich/govalidator@v9
(5/35) Wrote github.com/htcat/[email protected]
(6/35) Wrote github.com/k0kubun/[email protected]
(7/35) Wrote github.com/dgrijalva/[email protected]
(8/35) Wrote github.com/fatih/[email protected]
(9/35) Wrote github.com/knqyf263/go-cpe@master
(10/35) Wrote github.com/google/subcommands@master
(11/35) Wrote github.com/go-sql-driver/[email protected]
(12/35) Wrote github.com/inconshreveable/[email protected]
(13/35) Wrote github.com/go-stack/[email protected]
(14/35) Wrote github.com/go-redis/[email protected]
(15/35) Wrote github.com/cheggaaa/[email protected]
(16/35) Wrote github.com/labstack/[email protected]
(17/35) Wrote github.com/mattn/[email protected]
(18/35) Wrote github.com/mattn/[email protected]
(19/35) Wrote github.com/mattn/[email protected]
(20/35) Wrote github.com/lib/pq@master
(21/35) Wrote github.com/jinzhu/[email protected]
(22/35) Failed to write github.com/labstack/[email protected]
(23/35) Failed to write github.com/olekukonko/tablewriter@master
The authenticity of host 'github.com (192.30.255.113)' can't be established.
RSA key fingerprint is SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8.
RSA key fingerprint is MD5:16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48.
Are you sure you want to continue connecting (yes/no)? (24/35) Failed to write github.com/pkg/[email protected]
The authenticity of host 'github.com (192.30.255.113)' can't be established.
RSA key fingerprint is SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8.
RSA key fingerprint is MD5:16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48.
Are you sure you want to continue connecting (yes/no)? (25/35) Failed to write github.com/valyala/bytebufferpool@master
The authenticity of host 'github.com (192.30.255.113)' can't be established.
RSA key fingerprint is SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8.
RSA key fingerprint is MD5:16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48.
Are you sure you want to continue connecting (yes/no)? (26/35) Failed to write github.com/valyala/fasttemplate@master
(27/35) Failed to write golang.org/x/crypto@master
(28/35) Failed to write golang.org/x/sys@master
(29/35) Failed to write google.golang.org/[email protected]
(30/35) Failed to write gopkg.in/VividCortex/[email protected]
(31/35) Failed to write gopkg.in/cheggaaa/[email protected]
(32/35) Failed to write gopkg.in/fatih/[email protected]
(33/35) Failed to write gopkg.in/mattn/[email protected]
(34/35) Failed to write gopkg.in/mattn/[email protected]
(35/35) Failed to write github.com/mattn/[email protected]
grouped write of manifest, lock and vendor: error while writing out vendor tree: failed to write dep tree: failed to export github.com/labstack/echo: fatal: failed to unpack tree object 6d227dfea4d2e52cb76856120b3c17f758139b4e
: exit status 128
make: *** [dep] エラー 1

fetchnvd --years 2005 2008 2010 2011 Stops halfway

The text you enter will no longer appear at the prompt.

Hi.
The text you enter will no longer appear at the prompt.
"stty sane" returned to original.

Version is using this.
go-cve-dictionary v0.1.1 fde7146

$ sudo /opt/go/bin/goval-dictionary fetch-redhat 6 7
[Feb 15 10:24:49] INFO Fetching... https://www.redhat.com/security/data/oval/com.redhat.rhsa-RHEL6.xml.bz2

com.redhat.rhsa-RHEL6.xml.bz2: 882.52 KiB / 882.52 KiB [==============================================] 100.00%[Feb 15 10:24:51] INFO Finished to fetch OVAL definitions.
com.redhat.rhsa-RHEL6.xml.bz2: 882.52 KiB / 882.52 KiB [==============================================] 100.00%[Feb 15 10:24:51] INFO Fetched: https://www.redhat.com/security/data/oval/com.redhat.rhsa-RHEL7.xml.bz2
[Feb 15 10:24:51] INFO 579 OVAL definitions
[Feb 15 10:24:51] INFO Skip redhat 7 (Same Timestamp)
com.redhat.rhsa-RHEL6.xml.bz2: 882.52 KiB / 882.52 KiB [==============================================] 100.00%[Feb 15 10:24:52] INFO Fetched: https://www.redhat.com/security/data/oval/com.redhat.rhsa-RHEL6.xml.bz2
[Feb 15 10:24:52] INFO 1226 OVAL definitions
[Feb 15 10:24:52] INFO Skip redhat 6 (Same Timestamp)

Can't run 'make install'

go get -u github.com/golang/dep/...

runtime

/usr/local/go/src/runtime/os_linux.go:367: rt_sigaction redeclared in this block
previous declaration at /usr/local/go/src/runtime/cgo_sigaction.go:20
/usr/local/go/src/runtime/signal_sighandler.go:15: crashing redeclared in this block
previous declaration at /usr/local/go/src/runtime/signal_amd64x.go:39
/usr/local/go/src/runtime/signal_sighandler.go:28: sighandler redeclared in this block
previous declaration at /usr/local/go/src/runtime/signal_amd64x.go:44
/usr/local/go/src/runtime/sizeclasses.go:75: _MaxSmallSize redeclared in this block
previous declaration at /usr/local/go/src/runtime/malloc.go:124
/usr/local/go/src/runtime/sizeclasses.go:79: _NumSizeClasses redeclared in this block
previous declaration at /usr/local/go/src/runtime/malloc.go:121
/usr/local/go/src/runtime/sizeclasses.go:80: _PageShift redeclared in this block
previous declaration at /usr/local/go/src/runtime/malloc.go:107
/usr/local/go/src/runtime/sizeclasses.go:83: class_to_size redeclared in this block
previous declaration at /usr/local/go/src/runtime/msize.go:49
/usr/local/go/src/runtime/sizeclasses.go:84: class_to_allocnpages redeclared in this block
previous declaration at /usr/local/go/src/runtime/msize.go:50
/usr/local/go/src/runtime/sizeclasses.go:86: divMagic redeclared in this block
previous declaration at /usr/local/go/src/runtime/msize.go:225
/usr/local/go/src/runtime/sizeclasses.go:93: class_to_divmagic redeclared in this block
previous declaration at /usr/local/go/src/runtime/msize.go:51
/usr/local/go/src/runtime/sizeclasses.go:93: too many errors
GNUmakefile:28: recipe for target 'dep' failed
make: *** [dep] Error 2

CPENames matching error `Malformed constraint: <= 12.1x46:d72`

How to reproduce

config.toml

  [servers.juniper]
    cpeNames = [
    "cpe:/o:juniper:junos:10.1",
    ]
    type = "pseudo"

Error

[Sep 13 12:38:24] ERROR [localhost] Failed to detect vulns of [cpe:/o:juniper:junos:10.1]: Malformed constraint:  <= 12.1x46:d72

Error encountered when make install

dear author! I got an error when i followed your instruction. The error is as follow

master: unable to deduce repository and source type for "golang.org/x/sys/unix": unable to read metadata: unable to fetch raw metadata: failed HTTP request to URL "http://golang.org/x/sys/unix?go-get=1": Get http://golang.org/x/sys/unix?go-get=1: dial tcp 216.239.37.1:80: i/o timeout
break-out-specials: Could not introduce github.com/Sirupsen/logrus@break-out-specials, as it is not allowed by constraint master from project github.com/go-cve-dictionary.

how can i solve that?

crash while fetching modified nvd data

Command used:
go-cve-dictionary fetchnvd -modified

error encountered:

0 / 1 [--------------------------------------------------------------------------------------------------------------------------------------------------------------------] 0.00%[Sep 13 04:49:40] INFO Fetching... https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-modified.xml.gz
0 / 1 [--------------------------------------------------------------------------------------------------------------------------------------------------------------------] 0.00%panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x68 pc=0x688382]

goroutine 7 [running]:
compress/gzip.(*Reader).Close(0x0, 0xa718e0, 0xc42018d070)
/usr/local/go/src/compress/gzip/gunzip.go:292 +0x22
github.com/kotakanbe/go-cve-dictionary/nvd.fetchFeedFile(0xc42002cc30, 0x44, 0xc420042f50, 0x1, 0x1, 0xa718e0, 0xc420b36110)
/home/lpsupport/go/src/github.com/kotakanbe/go-cve-dictionary/nvd/nvd.go:173 +0x6b7
github.com/kotakanbe/go-cve-dictionary/nvd.fetchFeedFileConcurrently.func2()
/home/lpsupport/go/src/github.com/kotakanbe/go-cve-dictionary/nvd/nvd.go:119 +0x10d
github.com/kotakanbe/go-cve-dictionary/util.GenWorkers.func1(0xc4200767e0)
/home/lpsupport/go/src/github.com/kotakanbe/go-cve-dictionary/util/util.go:15 +0x3b
created by github.com/kotakanbe/go-cve-dictionary/util.GenWorkers
/home/lpsupport/go/src/github.com/kotakanbe/go-cve-dictionary/util/util.go:13 +0x66

I even tried with fetching new git code: git fetch

Error still persist.

panic: runtime error: invalid memory address or nil pointer dereference

I cannot fetch cve database:

go-cve-dictionary server

[Apr 11 13:38:48]  INFO Opening DB. datafile: /root/cve.sqlite3
[Apr 11 13:38:48]  INFO Migrating DB
[Apr 11 13:38:48]  INFO Fetching vulnerability data from NVD because no NVD data found in DB.
[Apr 11 13:38:48]  INFO Fetching... http://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2003.xml.gz
[Apr 11 13:38:48]  INFO Fetching... http://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2002.xml.gz
[Apr 11 13:38:48]  INFO Fetching... http://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2004.xml.gz
 0 / 15 [---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------]   0.00%[Apr 11 13:38:48]  INFO Fetching... http://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2006.xml.gz
[Apr 11 13:38:48]  INFO Fetching... http://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2005.xml.gz
[Apr 11 13:38:48]  INFO Fetching... http://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2007.xml.gz
[Apr 11 13:38:48]  INFO Fetching... http://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2009.xml.gz
[Apr 11 13:38:48]  INFO Fetching... http://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2008.xml.gz
[Apr 11 13:38:48]  INFO Fetching... http://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2010.xml.gz
[Apr 11 13:38:48]  INFO Fetching... http://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2011.xml.gz
[Apr 11 13:38:48]  INFO Fetching... http://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2012.xml.gz
[Apr 11 13:38:48]  INFO Fetching... http://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2013.xml.gz
[Apr 11 13:38:48]  INFO Fetching... http://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2014.xml.gz
[Apr 11 13:38:48]  INFO Fetching... http://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2015.xml.gz
[Apr 11 13:38:48]  INFO Fetching... http://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2016.xml.gz
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x504835]

goroutine 26 [running]:
panic(0x96e9c0, 0xc82000e100)
        /usr/local/go/src/runtime/panic.go:464 +0x3e6
github.com/kotakanbe/go-cve-dictionary/nvd.fetchFeedFile(0xc820263200, 0x3f, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /root/go/src/github.com/kotakanbe/go-cve-dictionary/nvd/nvd.go:157 +0xd65
github.com/kotakanbe/go-cve-dictionary/nvd.fetchFeedFileConcurrently.func2()
        /root/go/src/github.com/kotakanbe/go-cve-dictionary/nvd/nvd.go:125 +0x1c5
github.com/kotakanbe/go-cve-dictionary/nvd.genWorkers.func1(0xc82005c300)
        /root/go/src/github.com/kotakanbe/go-cve-dictionary/nvd/nvd.go:96 +0x60
created by github.com/kotakanbe/go-cve-dictionary/nvd.genWorkers
        /root/go/src/github.com/kotakanbe/go-cve-dictionary/nvd/nvd.go:98 +0x6d
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x504835]

goroutine 27 [running]:
panic(0x96e9c0, 0xc82000e100)
        /usr/local/go/src/runtime/panic.go:464 +0x3e6
github.com/kotakanbe/go-cve-dictionary/nvd.fetchFeedFile(0xc8202631c0, 0x3f, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /root/go/src/github.com/kotakanbe/go-cve-dictionary/nvd/nvd.go:157 +0xd65
github.com/kotakanbe/go-cve-dictionary/nvd.fetchFeedFileConcurrently.func2()
        /root/go/src/github.com/kotakanbe/go-cve-dictionary/nvd/nvd.go:125 +0x1c5
github.com/kotakanbe/go-cve-dictionary/nvd.genWorkers.func1(0xc82005c300)
        /root/go/src/github.com/kotakanbe/go-cve-dictionary/nvd/nvd.go:96 +0x60
created by github.com/kotakanbe/go-cve-dictionary/nvd.genWorkers
        /root/go/src/github.com/kotakanbe/go-cve-dictionary/nvd/nvd.go:98 +0x6d

OS: RHEL 7.2

Any idea what is wrong and how to fix?

Repeatedly updating the CVE library

As shown

After repeated execution-modified, it is found that the data in the database has been added repeatedly. The expected effect should be that no data is added, and existing data is updated.

dep ensure make: dep: Command not found on Ubuntu 16.04

After typing make install, I got this error:

dep ensure
make: dep: Command not found
GNUmakefile:28: recipe for target 'dep' failed
make: *** [dep] Error 127

I know makedepend is available on the server, so I tried to cheat by symlinking /usr/bin/makedepend to /usr/bin/dep. It too failed with:

go get -u github.com/golang/dep/...
dep ensure
dep: error:  [mM]akefile is not present
GNUmakefile:28: recipe for target 'dep' failed
make: *** [dep] Error 1

Not sure what else to do, so welcome any ideas.

Parallel Download

This source code is likely to be helpful.
https://github.com/Code-Hex/pget

Question: multiple CVE_ID's on JVN db

Kanbe-san,
I was looking at the JVN sqlite file, when searching for a CVE_id I get the below,

sqlite> SELECT cve_detail_id,title,summary,jvn_link FROM jvns WHERE cve_id='CVE-2014-7169';
45401|QNAP QTS に OS コマンドインジェクションの脆弱性|QNAP Systems, Inc. が提供する QTS は、Turbo NAS 用の OS です。QTS には、GNU Bash の脆弱性 (JVNVU#97219505) に起因する OS コマンドインジェクションの脆弱性 (CWE-78) が存在します。

この脆弱性情報は、情報セキュリティ早期警戒パートナーシップに基づき下記の方が IPA に報告し、JPCERT/CC が開発者との調整を行いました。
報告者: 電気通信大学脇坂優樹氏|https://jvndb.jvn.jp/ja/contents/2014/JVNDB-2014-000126.html

But in reality this CVE should be reference to https://jvndb.jvn.jp/ja/contents/2014/JVNDB-2014-004399.html

I notice https://jvndb.jvn.jp/ja/contents/2014/JVNDB-2014-000126.html actually has multiple CVE's associated with it (including CVE-2014-7169), actually show as first one on the list.
When you pull the data into SQLite, are you only grabbing the first CVE?

Travis CI integration

#84

nil pointer while fetchnvd --years 2002

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x93caf7]

goroutine 1 [running]:
github.com/kotakanbe/go-cve-dictionary/fetcher/nvd/json.convert(0xc0081da000, 0x1a5a, 0x1f9c, 0xc00f286000, 0x48d, 0x4ec, 0x0, 0x0)
        /home/ubuntu/go/src/github.com/kotakanbe/go-cve-dictionary/fetcher/nvd/json/nvd.go:91 +0x447
github.com/kotakanbe/go-cve-dictionary/fetcher/nvd/json.FetchConvert(0xc0000c4480, 0x3, 0x4, 0x1, 0x1, 0xc0000c4480, 0x2, 0x4)
        /home/ubuntu/go/src/github.com/kotakanbe/go-cve-dictionary/fetcher/nvd/json/nvd.go:47 +0x34f
github.com/kotakanbe/go-cve-dictionary/commands.(*FetchNvdCmd).Execute(0xc00029a720, 0xc9ace0, 0xc000098000, 0xc00029a780, 0x0, 0x0, 0x0, 0x0)
        /home/ubuntu/go/src/github.com/kotakanbe/go-cve-dictionary/commands/fetchnvd.go:207 +0xbd9
github.com/google/subcommands.(*Commander).Execute(0xc0000b0000, 0xc9ace0, 0xc000098000, 0x0, 0x0, 0x0, 0xc0002b0748)
        /home/ubuntu/go/pkg/mod/github.com/google/[email protected]/subcommands.go:142 +0x2f9
github.com/google/subcommands.Execute(...)
        /home/ubuntu/go/pkg/mod/github.com/google/[email protected]/subcommands.go:420
main.main()
        /home/ubuntu/go/src/github.com/kotakanbe/go-cve-dictionary/main.go:46 +0x2ec```

Where can I change the table name

Hello new to golang, where can I change the table name? I want to change the table name according to my convinience.
Thanks

Feature: Return only cve id by CPE

I'd like something akin to the GetByCpeURI function, but that only returned the cveIDs. If you think that's a reasonable feature, I'm happy to write it.

Either with a new function, or with a functional parameter on GetByCpeURI.

Comments?

NVD Update URL not working due to 503 Error. Need to change the NVD update feed url.

The NVD service has been oberved to be unavailable for a day as on date of posting this issue.

Hence the go-cve-dictionary is not able to update the nvd feeds into its database from the site.

However, the same source can be obtained from NVD site with a different url pattern template :-
https://nvd.nist.gov/feeds/xml/cve/2.0/nvdcve-2.0-2018.xml.gz

If this "service not able" is a permanent issue, then the url for nvd feed can be changed to the above mentioned url pattern in the code branch for the update to work.

Thanks

Error while running "make install" command

OS: ubuntu 16.04

solve error: No versions of golang.org/x/sys/unix met constraints:
	master: Could not introduce golang.org/x/sys/unix@master, as its subpackage golang.org/x/sys/unix does not contain usable Go code (*build.NoGoError).. Package is required by:
	github.com/Sirupsen/logrus@master
	github.com/mattn/[email protected]
ensure Solve(): No versions of golang.org/x/sys/unix met constraints:
	master: Could not introduce golang.org/x/sys/unix@master, as its subpackage golang.org/x/sys/unix does not contain usable Go code (*build.NoGoError).. Package is required by:
	github.com/Sirupsen/logrus@master
	github.com/mattn/[email protected]
GNUmakefile:28: recipe for target 'dep' failed
make: *** [dep] Error 1

unable to `go get` on fedora

> go get github.com/kotakanbe/go-cve-dictionary
# github.com/kotakanbe/go-cve-dictionary
/usr/lib/golang/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
/usr/bin/ld: /tmp/go-link-954546738/000001.o: unrecognized relocation (0x2a) in section `.text'
/usr/bin/ld: final link failed: Bad value
collect2: error: ld returned 1 exit status

environment:

Fedora release 23 (Twenty Three)
kernel-4.4.8-300.fc23.x86_64
golang-1.6.2-1.fc24.x86_64
git-2.5.5-1.fc23.x86_64
gcc-5.3.1-6.fc23.x86_64
sqlite-3.11.0-3.fc23.x86_64

Clarification On NvdJSON Structures

Hello,

I have been looking through some of the code in this project and I had more of a question and not an issue to ask. Basically, I am trying to understand the purpose of the models.go file in comparison to say the nvd.go file. In both files you define a NvdJSON structure that is different and I am trying to understand why.

From my perspective I am more curious about the actual cve data and parsing that from a local .json file but I am just curious what the models.go file is for?

Thanks

Configurable log directory?

I've noticed that go-cve-dictionary stores logs in a hardcoded directory defined in this line: https://github.com/kotakanbe/go-cve-dictionary/blob/master/log/log.go#L21

How about making it possible to specify the log directory via a command-line argument?
e.g. go-cve-dictionary --logdir=/opt/vuls/var/log/go-cve-dictionary fetchnvd

Can not update after commit 9c4dc5db

When trying to update vuls I keep getting the error:

dep ensure -v
[snip the successful Wrote lines]
grouped write of manifest, lock and vendor: error while writing out vendor tree: failed to write dep tree: failed to export github.com/kotakanbe/go-cve-dictionary: fatal: failed to unpack tree object 9c4dc5db721c165bb3f10b2981449fd2c4572c1f
: exit status 128
make: *** [dep] Error 1

I thought it might be a bad update or something, so I completely deleted my $GOPATH/src folder and started over with the instructions from here: https://vuls.io/docs/en/install-manually-centos.html

Everything went smoothly until I got to the installation of vuls. Once again the 'make install' was failing out with the same error in the same place. Thinking maybe it was my system, I tried on a different one. Same error.

Any thoughts on how to get around it?

Remove NVD XML support

https://nvd.nist.gov/general/news/XML-Vulnerability-Feed-Retirement

Add list subcommand

Add list subcommand that shows the history of fetching.

Feed filename , LastModifiedAt, up-to-date or out-of-date
...

make install failure

Hi,
the make fails with these errors:
/usr/local/go/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
/usr/bin/ld: /tmp/go-link-754964633/000020.o: unrecognized relocation (0x2a) in section `.text'
/usr/bin/ld: final link failed: Bad value
collect2: error: ld returned 1 exit status

make: *** [install] Error 2

here the installation steps:
yum -y install sqlite git gcc
wget https://dl.google.com/go/go1.12.3.linux-amd64.tar.gz
tar -C /usr/local -xzf go1.12.3.linux-amd64.tar.gz
export GOPATH=$HOME/go
export PATH=$PATH:/usr/local/go/bin:$GOPATH/bin
mkdir -p $GOPATH/src/github.com/kotakanbe
cd $GOPATH/src/github.com/kotakanbe
git clone https://github.com/kotakanbe/go-cve-dictionary.git
cd go-cve-dictionary
make install

the env is:

go env

GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/root/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build126036410=/tmp/go-build -gno-record-gcc-switches"

install error with redis db, rdb

root@hostname:~/go# go get github.com/kotakanbe/go-cve-dictionary
# github.com/kotakanbe/go-cve-dictionary/db
src/github.com/kotakanbe/go-cve-dictionary/db/rdb.go:473:6: bar.SetWriter undefined (type *pb.ProgressBar has no field or method SetWriter)
src/github.com/kotakanbe/go-cve-dictionary/db/rdb.go:475:6: bar.SetWriter undefined (type *pb.ProgressBar has no field or method SetWriter)
src/github.com/kotakanbe/go-cve-dictionary/db/rdb.go:627:6: bar.SetWriter undefined (type *pb.ProgressBar has no field or method SetWriter)
src/github.com/kotakanbe/go-cve-dictionary/db/rdb.go:629:6: bar.SetWriter undefined (type *pb.ProgressBar has no field or method SetWriter)
src/github.com/kotakanbe/go-cve-dictionary/db/rdb.go:761:6: bar.SetWriter undefined (type *pb.ProgressBar has no field or method SetWriter)
src/github.com/kotakanbe/go-cve-dictionary/db/rdb.go:763:6: bar.SetWriter undefined (type *pb.ProgressBar has no field or method SetWriter)
src/github.com/kotakanbe/go-cve-dictionary/db/redis.go:213:6: bar.SetWriter undefined (type *pb.ProgressBar has no field or method SetWriter)
src/github.com/kotakanbe/go-cve-dictionary/db/redis.go:215:6: bar.SetWriter undefined (type *pb.ProgressBar has no field or method SetWriter)
src/github.com/kotakanbe/go-cve-dictionary/db/redis.go:269:6: bar.SetWriter undefined (type *pb.ProgressBar has no field or method SetWriter)
src/github.com/kotakanbe/go-cve-dictionary/db/redis.go:271:6: bar.SetWriter undefined (type *pb.ProgressBar has no field or method SetWriter)
src/github.com/kotakanbe/go-cve-dictionary/db/redis.go:271:6: too many errors

can you change what CVE database it uses?

hello,

is it possible to change the website where it gets the CVE's from? for example if I have this site
https://cve.circl.lu/ would I be able to change to get from that site?

Broken CPE parsing

INFO[11-07|08:17:30] Fetched... https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2014.json.gz
EROR[11-07|08:17:34] Failed to convert to model. cve: CVE-2014-5606, err: Error! cannot have unquoted ? embedded in formatted string.: Parse error

Offending CPE: cpe:2.3:a:disney:where\'s_my_perry?_free:1.5.1:*:*:*:*:android:*:*

Possibly a follow-up to #109 (and the move to cpe23)

run server crash

error stack
commit: ea3526b

{"time":"2017-06-22T16:21:15.710037245+08:00","level":"-","prefix":"echo","file":"asm_amd64.s","line":"515","message":"[\x1b[31mPANIC RECOVER\x1b[0m] runtime error: invalid memory address or nil pointer dereference goroutine 14 [running]:
github.com/kotakanbe/go-cve-dictionary/vendor/github.com/labstack/echo/middleware.RecoverWithConfig.func1.1.1(0xa12040, 0x1000, 0x2a4f0000, 0xea5960, 0xc4202ae090)
\t/home/chenqi59/gopath/src/github.com/kotakanbe/go-cve-dictionary/vendor/github.com/labstack/echo/middleware/recover.go:75 +0x134
panic(0x9675e0, 0xe8afb0)
\t/home/chenqi59/.goenv/versions/1.8.0/src/runtime/panic.go:489 +0x2cf
github.com/kotakanbe/go-cve-dictionary/db.(*RDBDriver).Get(0xc4203161c0, 0xc42011498a, 0xd, 0xc42011498a)
\t/home/chenqi59/gopath/src/github.com/kotakanbe/go-cve-dictionary/db/rdb.go:105 +0x287
github.com/kotakanbe/go-cve-dictionary/server.getCve.func1(0xea5960, 0xc4202ae090, 0xc420591a00, 0x486612)
\t/home/chenqi59/gopath/src/github.com/kotakanbe/go-cve-dictionary/server/server.go:65 +0xb5
github.com/kotakanbe/go-cve-dictionary/vendor/github.com/labstack/echo.(*Echo).add.func1(0xea5960, 0xc4202ae090, 0xecfc00, 0xc420171d40)
\t/home/chenqi59/gopath/src/github.com/kotakanbe/go-cve-dictionary/vendor/github.com/labstack/echo/echo.go:475 +0x90
github.com/kotakanbe/go-cve-dictionary/vendor/github.com/labstack/echo/middleware.LoggerWithConfig.func2.1(0xea5960, 0xc4202ae090, 0x0, 0x0)
\t/home/chenqi59/gopath/src/github.com/kotakanbe/go-cve-dictionary/vendor/github.com/labstack/echo/middleware/logger.go:107 +0x145
github.com/kotakanbe/go-cve-dictionary/vendor/github.com/labstack/echo/middleware.RecoverWithConfig.func1.1(0xea5960, 0xc4202ae090, 0x0, 0x0)
\t/home/chenqi59/gopath/src/github.com/kotakanbe/go-cve-dictionary/vendor/github.com/labstack/echo/middleware/recover.go:82 +0xe1
github.com/kotakanbe/go-cve-dictionary/vendor/github.com/labstack/echo/middleware.LoggerWithConfig.func2.1(0xea5960, 0xc4202ae090, 0x0, 0x0)
\t/home/chenqi59/gopath/src/github.com/kotakanbe/go-cve-dictionary/vendor/github.com/labstack/echo/middleware/logger.go:107 +0x145
github.com/kotakanbe/go-cve-dictionary/vendor/github.com/labstack/echo.(*Echo).ServeHTTP.func1(0xea5960, 0xc4202ae090, 0xc420014260, 0x7f7b8c1de340)
\t/home/chenqi59/gopath/src/github.com/kotakanbe/go-cve-dictionary/vendor/github.com/labstack/echo/echo.go:556 +0x166
github.com/kotakanbe/go-cve-dictionary/vendor/github.com/labstack/echo.(*Echo).ServeHTTP(0xc4200a1e40, 0xea3160, 0xc42033a440, 0xea1a80, 0xc4202820c0)
\t/home/chenqi59/gopath/src/github.com/kotakanbe/go-cve-dictionary/vendor/github.com/labstack/echo/echo.go:565 +0x1f4
github.com/kotakanbe/go-cve-dictionary/vendor/github.com/labstack/echo/engine/standard.(*Server).ServeHTTP(0xc4200f5320, 0xe9e720, 0xc4202ca1c0, 0xc4201e2c00)
\t/home/chenqi59/gopath/src/github.com/kotakanbe/go-cve-dictionary/vendor/github.com/labstack/echo/engine/standard/server.go:156 +0x36b
net/http.serverHandler.ServeHTTP(0xc4200a1ef0, 0xe9e720, 0xc4202ca1c0, 0xc4201e2c00)
\t/home/chenqi59/.goenv/versions/1.8.0/src/net/http/server.go:2568 +0x92
net/http.(*conn).serve(0xc42027c780, 0xe9ee20, 0xc42033a600)
\t/home/chenqi59/.goenv/versions/1.8.0/src/net/http/server.go:1825 +0x612
created by net/http.(*Server).Serve
\t/home/chenqi59/.goenv/versions/1.8.0/src/net/http/server.go:2668 +0x2ce

goroutine 1 [IO wait]:
net.runtime_pollWait(0x7f7b8c1de280, 0x72, 0xe9b2e0)
\t/home/chenqi59/.goenv/versions/1.8.0/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc4200b9568, 0x72, 0xe96fa8, 0xc420114920)
\t/home/chenqi59/.goenv/versions/1.8.0/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc4200b9568, 0xffffffffffffffff, 0x0)
\t/home/chenqi59/.goenv/versions/1.8.0/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc4200b9500, 0x0, 0xe99ca0, 0xc420114920)
\t/home/chenqi59/.goenv/versions/1.8.0/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc42000e718, 0xc42027c800, 0x9590e0, 0xffffffffffffffff)
\t/home/chenqi59/.goenv/versions/1.8.0/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).AcceptTCP(0xc42000e718, 0xc420499900, 0xc420499908, 0xc4204998f8)
\t/home/chenqi59/.goenv/versions/1.8.0/src/net/tcpsock.go:215 +0x49
github.com/kotakanbe/go-cve-dictionary/vendor/github.com/labstack/echo/engine/standard.tcpKeepAliveListener.Accept(0xc42000e718, 0x"}

labstack/echo upstream API changes

Hi,

I tried to install vuls and it seems go-cve-dictionary doesn't build properly. My Golang foo is minimal, but it seems this error:

root@vuls:~# go get -u github.com/kotakanbe/go-cve-dictionary
go/src/github.com/kotakanbe/go-cve-dictionary/server/server.go:39: undefined:  middleware.LoggerFromConfig

occurs due to changes introduced in this commit.

I manually changed LoggerFromConfig to LoggerWithConfig in server.go source and it does build and seems to work ok.

Due to my currently low Go skills I am not sure if there are any other consequences (i.e. if only the naming changed in upstream) for go-cve-dictionary so I think it's best if I won't be sending a pull request.

fetch update

go-cve-dictionary v0.2.1 18cdbd1

Even if updating with fetch does not increase the size of the dictionary file of cve.sqlite3, is not update properly reflected?

Support NVD JSON v1.0 schema

https://nvd.nist.gov/vuln/Data-Feeds/JSON-feed-changelog

1.0 - 10/30/2018

All paths for referenced schemas updated to ../1.0/.. 
https://scap.nist.gov/schema/nvd/feed/1.0/nvd_cve_feed_json_1.0.schema
https://csrc.nist.gov/schema/nvd/feed/1.0/CVE_JSON_4.0_min.schema
https://csrc.nist.gov/schema/nvd/feed/1.0/cvss-v2.0.json
https://csrc.nist.gov/schema/nvd/feed/1.0/cvss-v3.0.json
In the CVSS v2.0 section, the "vectorString" property is no longer encapsulated in parenthesis ()
Changed "minItems" to 0 for vendor_data, problemtype_data, description, reference_data, and description_data 
Fixed an issue where "version_affected" was not being populated in certain circumstances
Added a new boolean property "acInsufInfo": {"type": "boolean"} to the "baseMetricV2" section
"cpe" property in the configuration section is now named "cpe_match"
Added optional array "cpe_name" property to schema for future support
No longer populate the "cpe2.2Uri", however, it remains in the schema

#107

Unable to handle NVD's CVSSv3.1 data

It seems NVD starting to use CVSSv3.1. But go-cve-dictionary cannot handle and store to the DB.

I found the problem in "CVE-2019-16056".
https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-recent.json.gz contains following fragment.

"baseMetricV3" : {
  "cvssV3" : {
    "version" : "3.1",
    "vectorString" : "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N",
    "attackVector" : "NETWORK",
    "attackComplexity" : "LOW",
    "privilegesRequired" : "NONE",
    "userInteraction" : "NONE",
    "scope" : "UNCHANGED",
    "confidentialityImpact" : "HIGH",
    "integrityImpact" : "NONE",
    "availabilityImpact" : "NONE",
    "baseScore" : 7.5,
    "baseSeverity" : "HIGH"
  },
  "exploitabilityScore" : 3.9,
  "impactScore" : 3.6
}

But go-cve-dictionary returns as follows

"Cvss3": {
  "VectorString": "",
  "AttackVector": "",
  "AttackComplexity": "",
  "PrivilegesRequired": "",
  "UserInteraction": "",
  "Scope": "",
  "ConfidentialityImpact": "",
  "IntegrityImpact": "",
  "AvailabilityImpact": "",
  "BaseScore": 0,
  "BaseSeverity": "",
  "ExploitabilityScore": 0,
  "ImpactScore": 0
}

install fails on dependencies

make install
go get -u github.com/golang/dep/...

github.com/golang/dep/internal/gps

../../golang/dep/internal/gps/constraints.go:334: undefined: sort.SliceStable
../../golang/dep/internal/gps/constraints.go:353: undefined: sort.SliceStable
../../golang/dep/internal/gps/lock.go:55: undefined: sort.SliceIsSorted
../../golang/dep/internal/gps/lock.go:62: undefined: sort.Slice
make: *** [dep] Error 2

Slow download speed on ARM Debian on Beaglebone Black

I ran for i in seq 2002 $(date +"%Y"); do go-cve-dictionary fetchnvd -years $i; done on Beaglebone black containing Debian ARM and compared it with x86_64 Kali Linux VM. The earlier was very very slow even though I am on high speed LAN connection.

The connection kept pausing. Unsure if it is a hardware or software issue so I thought I should just report for reference

There is some wrong with my installing go-cve-dictionary About go get git error code 128

This is not Project Issue, More for A Getting Help for Golang Use with this project.

I have tried to add some cfg like this but no use ..

git config --global user.name "xxx"
git config --global user.email "[email protected]"
git config http.sslVerify "false"
git config --global http.postBuffer 524288000

This is the all showing code with my install

[root@localhost go-cve-dictionary]# make install 
go get -u github.com/golang/dep/...
dep ensure -v
# Gopkg.lock is out of sync with Gopkg.toml and project imports:
github.com/kotakanbe/go-cve-dictionary/commands: imported or required, but missing from Gopkg.lock's input-imports
github.com/kotakanbe/go-cve-dictionary/config: imported or required, but missing from Gopkg.lock's input-imports
github.com/kotakanbe/go-cve-dictionary/db: imported or required, but missing from Gopkg.lock's input-imports
github.com/kotakanbe/go-cve-dictionary/fetcher: imported or required, but missing from Gopkg.lock's input-imports
github.com/kotakanbe/go-cve-dictionary/fetcher/jvn/xml: imported or required, but missing from Gopkg.lock's input-imports
github.com/kotakanbe/go-cve-dictionary/fetcher/nvd: imported or required, but missing from Gopkg.lock's input-imports
github.com/kotakanbe/go-cve-dictionary/fetcher/nvd/json: imported or required, but missing from Gopkg.lock's input-imports
github.com/kotakanbe/go-cve-dictionary/fetcher/nvd/xml: imported or required, but missing from Gopkg.lock's input-imports
github.com/kotakanbe/go-cve-dictionary/log: imported or required, but missing from Gopkg.lock's input-imports
github.com/kotakanbe/go-cve-dictionary/models: imported or required, but missing from Gopkg.lock's input-imports
github.com/kotakanbe/go-cve-dictionary/server: imported or required, but missing from Gopkg.lock's input-imports
github.com/kotakanbe/go-cve-dictionary/util: imported or required, but missing from Gopkg.lock's input-imports

Root project is "github.com/vulslib/go-cve-dictionary"
 13 transitively valid internal packages
 32 external packages imported from 15 projects
(0)   ✓ select (root)
(1)	? attempt github.com/asaskevich/govalidator with 1 pkgs; at least 1 versions to try
(1)	    try github.com/asaskevich/govalidator@v9
(1)	✓ select github.com/asaskevich/govalidator@v9 w/1 pkgs
(2)	? attempt github.com/fatih/color with 1 pkgs; at least 1 versions to try
(2)	    try github.com/fatih/[email protected]
(2)	✓ select github.com/fatih/[email protected] w/1 pkgs
(3)	? attempt github.com/cheggaaa/pb with 1 pkgs; at least 1 versions to try
(3)	    try github.com/cheggaaa/[email protected]
(3)	✓ select github.com/cheggaaa/[email protected] w/1 pkgs
(4)	? attempt github.com/jinzhu/gorm with 4 pkgs; at least 1 versions to try
(4)	    try github.com/jinzhu/[email protected]
(4)	✓ select github.com/jinzhu/[email protected] w/4 pkgs
(5)	? attempt github.com/go-sql-driver/mysql with 1 pkgs; at least 1 versions to try
(5)	    try github.com/go-sql-driver/[email protected]
(5)	✓ select github.com/go-sql-driver/[email protected] w/1 pkgs
(6)	? attempt github.com/google/subcommands with 1 pkgs; at least 1 versions to try
(6)	    try github.com/google/subcommands@master
(6)	✓ select github.com/google/subcommands@master w/1 pkgs
(7)	? attempt github.com/go-redis/redis with 1 pkgs; at least 1 versions to try
(7)	    try github.com/go-redis/[email protected]
(7)	✓ select github.com/go-redis/[email protected] w/8 pkgs
(8)	? attempt github.com/kotakanbe/go-cve-dictionary with 12 pkgs; 11 versions to try
(8)	    try github.com/kotakanbe/[email protected]
(8)	✓ select github.com/kotakanbe/[email protected] w/12 pkgs
(9)	? attempt github.com/hashicorp/go-version with 1 pkgs; at least 1 versions to try
(9)	    try github.com/hashicorp/go-version@master
(9)	✓ select github.com/hashicorp/go-version@master w/1 pkgs
(10)  ? attempt github.com/k0kubun/pp with 1 pkgs; at least 1 versions to try
(10)      try github.com/k0kubun/[email protected]
(10)  ✓ select github.com/k0kubun/[email protected] w/1 pkgs
(11)  ? attempt github.com/knqyf263/go-cpe with 3 pkgs; at least 1 versions to try
(11)      try github.com/knqyf263/go-cpe@master
(11)  ✓ select github.com/knqyf263/go-cpe@master w/3 pkgs
(12)  ? attempt github.com/labstack/echo with 2 pkgs; at least 1 versions to try
(12)      try github.com/labstack/[email protected]
(12)  ✓ select github.com/labstack/[email protected] w/2 pkgs
(13)  ? attempt github.com/dgrijalva/jwt-go with 1 pkgs; at least 1 versions to try
(13)      try github.com/dgrijalva/[email protected]
(13)  ✓ select github.com/dgrijalva/[email protected] w/1 pkgs
(14)  ? attempt github.com/olekukonko/tablewriter with 1 pkgs; at least 1 versions to try
(14)      try github.com/olekukonko/tablewriter@master
(14)  ✓ select github.com/olekukonko/tablewriter@master w/1 pkgs
(15)  ? attempt github.com/htcat/htcat with 1 pkgs; at least 1 versions to try
(15)      try github.com/htcat/[email protected]
(15)  ✓ select github.com/htcat/[email protected] w/1 pkgs
(16)  ? attempt github.com/labstack/gommon with 4 pkgs; at least 1 versions to try
(16)      try github.com/labstack/[email protected]
(16)  ✓ select github.com/labstack/[email protected] w/4 pkgs
(17)  ? attempt github.com/jinzhu/inflection with 1 pkgs; at least 1 versions to try
(17)      try github.com/jinzhu/inflection@master
(17)  ✓ select github.com/jinzhu/inflection@master w/1 pkgs
(18)  ? attempt github.com/mattn/go-runewidth with 1 pkgs; at least 1 versions to try
(18)      try github.com/mattn/[email protected]
(18)  ✓ select github.com/mattn/[email protected] w/1 pkgs
(19)  ? attempt github.com/pkg/errors with 1 pkgs; at least 1 versions to try
(19)      try github.com/pkg/[email protected]
(19)  ✓ select github.com/pkg/[email protected] w/1 pkgs
(20)  ? attempt github.com/mattn/go-sqlite3 with 1 pkgs; at least 1 versions to try
(20)      try github.com/mattn/[email protected]
(20)  ✗   failed to fetch source for https://github.com/mattn/go-sqlite3: unable to get repository: 正克隆到 '/root/go/pkg/dep/sources/https---github.com-mattn-go--sqlite3'...
(20)    POST git-upload-pack (gzip 1192 to 639 bytes)
(20)    remote: Enumerating objects: 49, done.
remote: Counting objects: 100% (49/49), done.
remote: Compressing objects: 100% (31/31), done.
error: RPC failed; result=18, HTTP code = 200.00 KiB/s   
(20)    fatal: The remote end hung up unexpectedly
(20)    fatal: 过早的文件结束符（EOF）
(20)    fatal: index-pack failed
(20)    : command failed: [git clone --recursive -v --progress https://github.com/mattn/go-sqlite3 /root/go/pkg/dep/sources/https---github.com-mattn-go--sqlite3]: exit status 128
(20)      try github.com/mattn/[email protected]
(20)  ✗   failed to fetch source for https://github.com/mattn/go-sqlite3: unable to get repository: 正克隆到 '/root/go/pkg/dep/sources/https---github.com-mattn-go--sqlite3'...
(20)    POST git-upload-pack (gzip 1192 to 639 bytes)
(20)    remote: Enumerating objects: 49, done.
remote: Counting objects: 100% (49/49), done.
remote: Compressing objects: 100% (31/31), done.
error: RPC failed; result=18, HTTP code = 200.00 KiB/s   
(20)    fatal: The remote end hung up unexpectedly
(20)    fatal: 过早的文件结束符（EOF）
(20)    fatal: index-pack failed
(20)    : command failed: [git clone --recursive -v --progress https://github.com/mattn/go-sqlite3 /root/go/pkg/dep/sources/https---github.com-mattn-go--sqlite3]: exit status 128
(20)      try github.com/mattn/[email protected]
(20)  ✗   failed to fetch source for https://github.com/mattn/go-sqlite3: unable to get repository: 正克隆到 '/root/go/pkg/dep/sources/https---github.com-mattn-go--sqlite3'...
(20)    POST git-upload-pack (gzip 1192 to 639 bytes)
(20)    remote: Enumerating objects: 49, done.
remote: Counting objects: 100% (49/49), done.
remote: Compressing objects: 100% (31/31), done.
error: RPC failed; result=18, HTTP code = 200.00 KiB/s   
(20)    fatal: The remote end hung up unexpectedly
(20)    fatal: 过早的文件结束符（EOF）
(20)    fatal: index-pack failed
(20)    : command failed: [git clone --recursive -v --progress https://github.com/mattn/go-sqlite3 /root/go/pkg/dep/sources/https---github.com-mattn-go--sqlite3]: exit status 128
(20)      try github.com/mattn/[email protected]
(20)  ✓ select github.com/mattn/[email protected] w/1 pkgs
(21)  ? attempt github.com/valyala/fasttemplate with 1 pkgs; at least 1 versions to try
(21)      try github.com/valyala/fasttemplate@master
(21)  ✓ select github.com/valyala/fasttemplate@master w/1 pkgs
(22)  ? attempt github.com/mattn/go-isatty with 1 pkgs; at least 1 versions to try
(22)      try github.com/mattn/[email protected]
(22)  ✓ select github.com/mattn/[email protected] w/1 pkgs
(23)  ? attempt gopkg.in/mattn/go-colorable.v0 with 1 pkgs; at least 1 versions to try
(23)      try gopkg.in/mattn/[email protected]
(23)  ✓ select gopkg.in/mattn/[email protected] w/1 pkgs
(24)  ? attempt github.com/inconshreveable/log15 with 1 pkgs; at least 1 versions to try
(24)      try github.com/inconshreveable/[email protected]
(24)  ✓ select github.com/inconshreveable/[email protected] w/1 pkgs
(25)  ? attempt github.com/go-stack/stack with 1 pkgs; at least 1 versions to try
(25)      try github.com/go-stack/[email protected]
(25)  ✓ select github.com/go-stack/[email protected] w/1 pkgs
(26)  ? attempt github.com/lib/pq with 2 pkgs; at least 1 versions to try
(26)      try github.com/lib/pq@master
(26)  ✓ select github.com/lib/pq@master w/3 pkgs
(27)  ? attempt github.com/mattn/go-colorable with 1 pkgs; at least 1 versions to try
(27)      try github.com/mattn/[email protected]
(27)  ✓ select github.com/mattn/[email protected] w/1 pkgs
(28)    ← no more versions of golang.org/x/sys to try; begin backtrack
(27)  ← backtrack: no more versions of github.com/mattn/go-colorable to try
(26)  ← backtrack: no more versions of github.com/lib/pq to try
(25)  ← backtrack: no more versions of github.com/go-stack/stack to try
(24)  ← backtrack: no more versions of github.com/inconshreveable/log15 to try
(23)  ← backtrack: no more versions of gopkg.in/mattn/go-colorable.v0 to try
(22)  ← backtrack: no more versions of github.com/mattn/go-isatty to try
(21)  ← backtrack: no more versions of github.com/valyala/fasttemplate to try
(20)  ← backtrack: no more versions of github.com/mattn/go-sqlite3 to try
(19)  ← backtrack: no more versions of github.com/pkg/errors to try
(18)  ← backtrack: no more versions of github.com/mattn/go-runewidth to try
(17)  ← backtrack: no more versions of github.com/jinzhu/inflection to try
(16)  ← backtrack: no more versions of github.com/labstack/gommon to try
(15)  ← backtrack: no more versions of github.com/htcat/htcat to try
(14)  ← backtrack: no more versions of github.com/olekukonko/tablewriter to try
(13)  ← backtrack: no more versions of github.com/dgrijalva/jwt-go to try
(12)  ← backtrack: no more versions of github.com/labstack/echo to try
(11)  ← backtrack: no more versions of github.com/knqyf263/go-cpe to try
(10)  ← backtrack: no more versions of github.com/k0kubun/pp to try
(9)	← backtrack: no more versions of github.com/hashicorp/go-version to try
(8)	← backtrack: no more versions of github.com/kotakanbe/go-cve-dictionary to try
(7)	← backtrack: no more versions of github.com/go-redis/redis to try
(6)	← backtrack: no more versions of github.com/google/subcommands to try
(5)	← backtrack: no more versions of github.com/go-sql-driver/mysql to try
(4)	← backtrack: no more versions of github.com/jinzhu/gorm to try
(3)	← backtrack: no more versions of github.com/cheggaaa/pb to try
(2)	← backtrack: no more versions of github.com/fatih/color to try
(1)	← backtrack: no more versions of github.com/asaskevich/govalidator to try
  ✗ solving failed

Solver wall times by segment:
         b-list-pkgs: 6m58.539239175s
     b-source-exists:   53.666160219s
              b-gmal:    1.339633737s
     b-list-versions:    1.287272728s
  b-deduce-proj-root:     1.24541074s
             satisfy:      1.772314ms
         select-atom:      1.661675ms
            new-atom:      1.243902ms
            unselect:      1.217725ms
         select-root:       266.177µs
           backtrack:       119.869µs
               other:        50.904µs

  TOTAL: 7m56.084049165s

Solving failure: 
	(1) failed to list versions for https://go.googlesource.com/sys: fatal: unable to access 'https://go.googlesource.com/sys/': Peer's Certificate issuer is not recognized.
: exit status 128

make: *** [dep] 错误 1

unresolved dependencies

root@skinner1:~/go# mkdir -p $GOPATH/src/github.com/kotakanbe

root@skinner1:~/go# cd $GOPATH/src/github.com/kotakanbe

root@skinner1:~/go/src/github.com/kotakanbe# git clone https://github.com/kotakanbe/go-cve-dictionary.git
Cloning into 'go-cve-dictionary'...
remote: Counting objects: 1023, done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 1023 (delta 0), reused 2 (delta 0), pack-reused 1018
Receiving objects: 100% (1023/1023), 282.02 KiB | 867.00 KiB/s, done.
Resolving deltas: 100% (523/523), done.

root@skinner1:~/go/src/github.com/kotakanbe# cd go-cve-dictionary

root@skinner1:~/go/src/github.com/kotakanbe/go-cve-dictionary# make install
go get -u github.com/golang/dep/...
package github.com/golang/dep
	imports context: unrecognized import path "context" (import path does not begin with hostname)
GNUmakefile:28: recipe for target 'dep' failed
make: *** [dep] Error 1

root@skinner1:~/go/src/github.com/kotakanbe/go-cve-dictionary#

as you see i have a problem installing your code, it has "unresolved dependencies" (rather false ones).

go-cve-dictionary fetchnvd and fetchjvn has Notation collapse on EC2UserData

Hi.
I encountered notation collapse, On EC2 Instance's UserData.
Like this.

Premise

Installed Go accoding to https://vuls.io/docs/ja/install-manually-centos.html

here is code executed as UserData

mkdir -p $GOPATH/src/github.com/kotakanbe
cd $GOPATH/src/github.com/kotakanbe
git clone https://github.com/kotakanbe/go-cve-dictionary.git
cd go-cve-dictionary
make install

# cve
for i in `seq 2002 $(date +"%Y")`; do /home/vuls/vuls/bin/go-cve-dictionary fetchnvd -years $i; done
for i in `seq 1998 $(date +"%Y")`; do /home/vuls/vuls/bin/go-cve-dictionary fetchjvn -years $i; done

connect instance and run this command.

tail -f /var/log/cloud-init-output.log

environment

aws EC2
ami: ami-06cd52961ce9f0d85
- Amazon Linux 1

panic on JSON unmarshal of NVD

I'm getting a panic when trying to pull NVD from 2002:

DBUG[09-11|15:42:25] Opening DB (redis).
INFO[09-11|15:42:25] Fetching... https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2002.meta
INFO[09-11|15:42:25] Fetching... https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-modified.meta
INFO[09-11|15:42:25] Fetching... https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-recent.meta
INFO[09-11|15:42:26] Fetched... https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2002.meta
INFO[09-11|15:42:26] Fetched... https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-recent.meta
INFO[09-11|15:42:26] Fetched... https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-modified.meta
INFO[09-11|15:42:26]      Newly: https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2002.json.gz
INFO[09-11|15:42:26] Up to date: https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-recent.json.gz
INFO[09-11|15:42:26] Up to date: https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-modified.json.gz
INFO[09-11|15:42:26] Fetching... https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2002.json.gz
INFO[09-11|15:42:26] Fetched... https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2002.json.gz
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x90e5a7]

goroutine 1 [running]:
github.com/kotakanbe/go-cve-dictionary/fetcher/nvd/json.convert(0xc005218000, 0x1a5a, 0x1f9c, 0xc00da8e000, 0x48f, 0x4ec, 0x0, 0x0)
        /Users/nklauer/go/src/github.com/kotakanbe/go-cve-dictionary/fetcher/nvd/json/nvd.go:91 +0x447
github.com/kotakanbe/go-cve-dictionary/fetcher/nvd/json.FetchConvert(0xc0000bce10, 0x1, 0x1, 0x1, 0x1, 0xc0000bce10, 0x0, 0x1)
        /Users/nklauer/go/src/github.com/kotakanbe/go-cve-dictionary/fetcher/nvd/json/nvd.go:47 +0x34f
github.com/kotakanbe/go-cve-dictionary/commands.(*FetchNvdCmd).Execute(0xc000286720, 0xbb85c0, 0xc000080000, 0xc000286780, 0x0, 0x0, 0x0, 0x0)
        /Users/nklauer/go/src/github.com/kotakanbe/go-cve-dictionary/commands/fetchnvd.go:207 +0xbd9
github.com/google/subcommands.(*Commander).Execute(0xc00009c000, 0xbb85c0, 0xc000080000, 0x0, 0x0, 0x0, 0xc00028e908)
        /Users/nklauer/go/pkg/mod/github.com/google/[email protected]/subcommands.go:142 +0x2f9
github.com/google/subcommands.Execute(...)
        /Users/nklauer/go/pkg/mod/github.com/google/[email protected]/subcommands.go:420
main.main()
        /Users/nklauer/go/src/github.com/kotakanbe/go-cve-dictionary/main.go:46 +0x2ec

The version I have is a tweaked version, as I cannot cross-compile it on my Mac to Linux due to some issues with Sqlite. Since we aren't using SQLITE for our instance, I'm omitting that DB type entirely.

The error points to this line:

https://github.com/kotakanbe/go-cve-dictionary/blob/master/fetcher/nvd/json/nvd.go#L92

some cve not in database

Hi. I found CVE-2015-1328 not in the database. But there is one page about this vuln, http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-1328. Thanks.

Go-CVE-Dictionary FetchNVD Error

Hello,

Can you please tell me why I am getting this error when I try to download NVD CVEs locally. If I wget the file directly "https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2002.xml.gz" I don't get any errors. However when doing it as follows I get this error.

"go-cve-dictionary/unstable,now 0.1.1+git20180404.0.4ee71e8-1 amd64 [installed]"

root@REDACTED:/opt/download# for i in seq 2002 $(date +"%Y"); do go-cve-dictionary fetchnvd -years $i; done
0 / 1 [---------------------------------------------------------------------------------------------------------------------------------] 0.00%[Oct 31 11:57:44] INFO Fetching... https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2002.xml.gz
1 / 1 [==============================================================================================================================] 100.00% 1s
[Oct 31 11:57:45] ERROR Failed to fetch cve data from NVD. err: [HTTP error. errs: [], url: https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2002.xml.gz]

CVSS2 and CVSS3 data missing for some CVEs (e.g. CVE-2017-16844)

I'm running the latest docker image from Dockerhub.

I did go-cve-dictionary fetchnvd -last2y without errors

I'm seeing that some CVEs are missing score data, but that data appears on the NVD website.

e.g.

{
  "CveID": "CVE-2017-16844",
  "NvdJSON": {
    "CveID": "CVE-2017-16844",
    "Descriptions": [
      {
        "Lang": "en",
        "Value": "Heap-based buffer overflow in the loadbuf function in formisc.c in formail in procmail 3.22 allows remote attackers to cause a denial of service (application crash) or possibly execute arbitrary code via a crafted e-mail message because of a hardcoded realloc size, a different vulnerability than CVE-2014-3618."
      }
    ],
    "Cvss2": {
      "VectorString": "",
      "AccessVector": "",
      "AccessComplexity": "",
      "Authentication": "",
      "ConfidentialityImpact": "",
      "IntegrityImpact": "",
      "AvailabilityImpact": "",
      "BaseScore": 0,
      "Severity": "",
      "ExploitabilityScore": 0,
      "ImpactScore": 0,
      "ObtainAllPrivilege": false,
      "ObtainUserPrivilege": false,
      "ObtainOtherPrivilege": false,
      "UserInteractionRequired": false
    },
    "Cvss3": {
      "VectorString": "",
      "AttackVector": "",
      "AttackComplexity": "",
      "PrivilegesRequired": "",
      "UserInteraction": "",
      "Scope": "",
      "ConfidentialityImpact": "",
      "IntegrityImpact": "",
      "AvailabilityImpact": "",
      "BaseScore": 0,
      "BaseSeverity": "",
      "ExploitabilityScore": 0,
      "ImpactScore": 0
    },
    "Cwes": [
      {
        "CweID": "CWE-119"
      }
    ],
    "Cpes": [
      {
        "URI": "cpe:/a:procmail:procmail:3.22",
        "FormattedString": "cpe:2.3:a:procmail:procmail:3.22:*:*:*:*:*:*:*",
        "WellFormedName": "wfn:[part=\"a\", vendor=\"procmail\", product=\"procmail\", version=\"3\\.22\", update=ANY, edition=ANY, language=ANY, sw_edition=ANY, target_sw=ANY, target_hw=ANY, other=ANY]",
        "Part": "a",
        "Vendor": "procmail",
        "Product": "procmail",
        "Version": "3\\.22",
        "Update": "ANY",
        "Edition": "ANY",
        "Language": "ANY",
        "SoftwareEdition": "ANY",
        "TargetSW": "ANY",
        "TargetHW": "ANY",
        "Other": "ANY",
        "VersionStartExcluding": "",
        "VersionStartIncluding": "",
        "VersionEndExcluding": "",
        "VersionEndIncluding": "",
        "EnvCpes": []
      }
    ],
    "Affects": [
      {
        "Vendor": "procmail",
        "Product": "procmail",
        "Version": "3.22"
      }
    ],
    "References": [
      {
        "Source": "",
        "Link": "http://www.securitytracker.com/id/1039844"
      },
      {
        "Source": "",
        "Link": "https://access.redhat.com/errata/RHSA-2017:3269"
      },
      {
        "Source": "",
        "Link": "https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=876511"
      },
      {
        "Source": "",
        "Link": "https://lists.debian.org/debian-lts-announce/2017/11/msg00019.html"
      },
      {
        "Source": "",
        "Link": "https://www.debian.org/security/2017/dsa-4041"
      }
    ],
    "PublishedDate": "2017-11-16T15:29:00Z",
    "LastModifiedDate": "2018-02-04T02:29:00Z"
  }
}

Support nvd json 1.1

#137

Pull request welcome :)

Unable `go get` in seveal platforms

I tried to go get github.com/kotakanbe/go-cve-dictionary in CentOS 7.1 and MacOSX.
and it failed.

$ go get github.com/kotakanbe/go-cve-dictionary
package github.com/labstack/echo/engine/standard: cannot find package "github.com/labstack/echo/engine/standard" in any of:

After labstack/echo released v3.0.0, engine directory disappeared.

Could you fix the dependencies?

make install fails

I am unable to build this project from source. I am on Go 1.13.

 make install
go get -u github.com/golang/dep/...
go: finding github.com/sdboyer/constext latest
go: finding github.com/nightlyone/lockfile latest
go: finding golang.org/x/sys latest
go: finding golang.org/x/sync latest
# github.com/golang/dep/gps
../../../../pkg/mod/github.com/golang/[email protected]/gps/constraint.go:149:4: undefined: semver.Constraint
make: *** [dep] Error 2