Coder Social home page Coder Social logo

arcus-java-client's Introduction

arcus-java-client: Arcus Java Client CI License

This is a fork of spymemcached with the following modifications to support Arcus memcached cloud.

  • Collection data types
    • List: A doubly-linked list.
    • Set: An unordered set of unique data.
    • Map: An unordered set of <field, value>.
    • B+Tree: A B+Tree structure similar to sorted map.
  • ZooKeeper based clustering

JDK Requirements

Compatible with jdk version

  • runtime requirements : At least 1.6
  • build requirements : At least 1.8

Getting Started

The Maven artifact for arcus java client is in the central Maven repository. To use it, add the following dependency to your pom.xml.

<dependencies>
    <dependency>
        <groupId>com.navercorp.arcus</groupId>
        <artifactId>arcus-java-client</artifactId>
        <version>1.13.4</version>
    </dependency>
</dependencies>

Building

To build your own library, simply run the following maven command:

$ mvn clean install

# Test cases may not run properly if you do not already have memcached
# and ZooKeeper installed on the local machine.  To skip tests, use skipTests.

$ mvn clean install -DskipTests=true

Running Test Cases

Before running test cases, make sure to set up a local ZooKeeper and run an Arcus memcached instance. Several Arcus specific test cases assume that there is an Arcus instance running, along with ZooKeeper.

First, make a simple ZooKeeper configuration file. By default, tests assume ZooKeeper is running at localhost:2181.

$ cat test-zk.conf
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
dataDir=/home1/openarcus/zookeeper_data
# the port at which the clients will connect
clientPort=2181
maxClientCnxns=200

Second, create znodes for one memcached instance running at localhost:11211. ZooKeeper comes with a command line tool. The following script uses it to set up the directory structure.

$ cat setup-test-zk.bash

ZK_CLI="./zookeeper/bin/zkCli.sh"
ZK_ADDR="-server localhost:2181"

$ZK_CLI $ZK_ADDR create /arcus 0
$ZK_CLI $ZK_ADDR create /arcus/cache_list 0
$ZK_CLI $ZK_ADDR create /arcus/cache_list/test 0
$ZK_CLI $ZK_ADDR create /arcus/client_list 0
$ZK_CLI $ZK_ADDR create /arcus/client_list/test 0
$ZK_CLI $ZK_ADDR create /arcus/cache_server_mapping 0
$ZK_CLI $ZK_ADDR create /arcus/cache_server_log 0
$ZK_CLI $ZK_ADDR create /arcus/cache_server_mapping/127.0.0.1:11211 0
$ZK_CLI $ZK_ADDR create /arcus/cache_server_mapping/127.0.0.1:11211/test 0
$ZK_CLI $ZK_ADDR create /arcus/cache_server_mapping/127.0.0.1:11212 0
$ZK_CLI $ZK_ADDR create /arcus/cache_server_mapping/127.0.0.1:11212/test 0

Now start the ZooKeeper instance using the configuration above.

$ ZOOCFGDIR=$PWD ./zookeeper/bin/zkServer.sh start test-zk.conf

And, start the memcached instance.

$ /home1/openarcus/bin/memcached -E /home1/openarcus/lib/default_engine.so -p 11211 -z localhost:2181

Finally, run test cases.

$ mvn test
[...]
Results :

Tests run: 722, Failures: 0, Errors: 0, Skipped: 8

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3:17.308s
[INFO] Finished at: Thu Mar 06 13:42:58 KST 2014
[INFO] Final Memory: 9M/722M
[INFO] ------------------------------------------------------------------------

API Documentation

Please refer to Arcus Java Client User Guide for the detailed usage of Arcus java client.

Issues

If you find a bug, please report it via the GitHub issues page.

https://github.com/naver/arcus-java-client/issues

Arcus Contributors

In addition to those who had contributed to the original libmemcached, the following people at NAVER have contributed to arcus-java-client.

Chisu Yu (netspider) [email protected]; [email protected]
Hoonmin Kim (harebox) [email protected]; [email protected]
YeaSol Kim (ngleader) [email protected]; [email protected]
SeongHwa Ahn [email protected]; [email protected]
HyongYoub Kim [email protected]

License

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0

Patents

Arcus has patents on b+tree smget operation. Refer to PATENTS file in this directory to get the patent information.

Under the Apache License 2.0, a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable patent license is granted to any user for any usage. You can see the specifics on the grant of patent license in LICENSE file in this directory.

arcus-java-client's People

Contributors

aiceru avatar brido4125 avatar computerphilosopher avatar devhue avatar eunji6546 avatar gadimli93 avatar greenmonn avatar hjyun328 avatar hoonmin avatar hyongyoubkim avatar jhpark816 avatar jkjh0819 avatar jongwooo avatar jooho812 avatar minkikim89 avatar minwoojin avatar oliviarla avatar rudeus avatar seongminy avatar suhwanjang avatar supniverse avatar thjeong917 avatar uhm0311 avatar vcho1958 avatar voyageth avatar weirdjh avatar whchoi83 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

arcus-java-client's Issues

BopFindPositionWithGetTest 객체 package 경로 수정 필요

BopFindPositionWithGetTest.java test 객체의 위치가
코드 상에는 package net.spy.memcached.collection.btree 로 되어 있고,
실제로는 (src/test/manual/)net.spy.memcached.collection 경로에 존재합니다.

객체의 특성상 net.spy.memcached.collection.btree 경로 위치에 존재하는 것이 맞을 것 같습니다.

인터페이스 구현체의 interface 지정 이슈

CollectionUpsertOperationImpl의 interface 지정이 아래와 같습니다.
CollectionUpsertOperation interface를 지정하도록 수정하는 게 좋을 것 같습니다.

참고 사항으로, CollectionStoreOperation와 CollectionUpsertOperation가 동일한 interface이기 때문에,
동작에는 문제가 없습니다.

public class CollectionUpsertOperationImpl extends OperationImpl implements
        CollectionStoreOperation {
    . . .
}

Bop decr 수행의 오류

ArcusClientPool.java 코드를 보면, 아래와 같습니다.
확인하시고, Bop decr 관련 전체 코드를 확인하고, 수정이 필요합니다.

	@Override
	public CollectionFuture<Long> asyncBopDecr(String key, long subkey, int by) {
		return this.getClient().asyncBopIncr(key, subkey, by);
	}

	@Override
	public CollectionFuture<Long> asyncBopDecr(String key, byte[] subkey, int by) {
		return this.getClient().asyncBopIncr(key, subkey, by);
	}

	@Override
	public CollectionFuture<Long> asyncBopDecr(String key, long subkey,
			int by, long initial, byte[] eFlag) {
		return this.getClient().asyncBopIncr(key, subkey, by, initial, eFlag);
	}

	@Override
	public CollectionFuture<Long> asyncBopDecr(String key, byte[] subkey,
			int by, long initial, byte[] eFlag) {
		return this.getClient().asyncBopIncr(key, subkey, by, initial, eFlag);
	}

[개선] bulk service 메인 로직 최적화

기존 bulk service 메인 로직은 2가지 이슈가 있습니다.

  • Readability 떨어짐.
  • 성능을 최적화 가능한 부분 존재
    • 기존 방식은 "N개 연산 단위로 연산 처리를 반복"하는 방법으로,
      N개 연산의 처리가 완료될 때까지 next 연산을 보내지 않는다.
    • "항상 N개 연산이 동시 처리"되도록 변경하면 나은 성능을 도출할 수 있다.

이를 위해, 변경한 로직은 다음과 같다.

    public Map<String, CollectionOperationStatus> call() throws Exception {
      int numActiveOperations = 0;
      int posResponseReceived = 0;

      for (int pos = fromIndex; isRunnable() && pos <= toIndex; pos++) {
        try {
          if (isRunnable()) {
            future.add(pos, processItem(pos));
          }
        } catch (IllegalStateException e) {
          if (Thread.currentThread().isInterrupted()) {
            break;
          } else {
            throw e;
          }
        }
        numActiveOperations++;

        if (numActiveOperations >= DEFAULT_LOOP_LIMIT) {
          awaitProcessResult(posResponseReceived);
          posResponseReceived++;
          numActiveOperations--;
        }
      }

      while (numActiveOperations > 0) {
        awaitProcessResult(posResponseReceived);
        posResponseReceived++;
        numActiveOperations--;
      }
      return errorList;
    }

@minkikim89
검토하고, 문제가 없다면 반영해 주세요.

MutatorOperation에서 TYPE_MISMATCH status 확인 코드 필요.

collection item에 대해 incr or decr 연산을 수행하면, TYPE_MISMATCH response가 전달된다.
이 경우의 status matching 로직의 추가가 필요하다.
그 외에도 의도하지 않은 status가 올 수 있음을 가정하여 구현되어야 한다.

bulkServiceLoopLimit API명에 대한 변경 검토

현재 bulkService는 코드가 변경 되어 더 이상 loop 형태로 데이터를 전송하지 않는다.
bulkServiceLoopLimit 변수 값은 이전과 다르게 한번에 보내는 연산의 양이 너무 많이 쌓이지 않도록 조절하는 역할만을 한다.
이에, 해당 변수명을 수정하는 것을 검토하고, 변수명이 변할 경우 API명 변경도 고려해야한다.

Switchover move operation 동작 Operation 순서 검토

switchover 의 전체 동작은 아래와 같다.

  1. replication group change Role
  2. move operations. Master to Slave
  3. Master node reconnect

이 중 1번과 2번. change role이 완료되고 move operations가 수행되기 전 응용의 작업 thread가
role change가 된 new master의 inputQ에 새로운 operation을 삽입할 수 있다.
결국, switchover move operation이 수행되면서 operation 순서가 보장되지 않을 수 있다.

old Master Node Operation Queue

  • RQ : Op1
  • WQ: Op2, Op3, Op4
  • IQ : Op5, Op6

move operation 도중 new Master InputQ에 insert

new Master Node Operation Queue

  • RQ:
  • WQ:
  • Iq: Op7, Op8, Op1, Op2, Op3, Op4, Op5, Op6

순서를 보장할 수 있는 방법이 필요 할 것으로 보인다.

@hjyun328
현재 작업중인 #233 와 관련된 이슈로,
같이 검토가 필요할 것 같습니다.

아커스키를 한글로 지정하고 attribute조회할때 에러 발생

키를 한글로(7자리이상) 하고 attributes 조회할때

java.nio.BufferOverflowException
at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:189)
at java.nio.ByteBuffer.put(ByteBuffer.java:859)
at net.spy.memcached.protocol.ascii.OperationImpl.setArguments(OperationImpl.java:105)
at net.spy.memcached.protocol.ascii.GetAttrOperationImpl.initialize(GetAttrOperationImpl.java:89)

이런 에러가 나옵니다.

ByteBuffer allocate할때 size설정이 잘못된거 같습니다.

GetAttrOperationImpl.java
@OverRide
public void initialize() {
int size = CMD.length() + key.length() + 16;
ByteBuffer bb = ByteBuffer.allocate(size);
setArguments(bb, CMD, key);
....
}

key.length로 계산하다보니 한글일경우 문제가 생기는거 같습니다.

수정이 가능할까요?

사용예)

    CollectionFuture<CollectionAttributes> future = null;
    CollectionAttributes result = null;

    future = this.arcusClient.asyncGetAttr(arcusKey);

Switchover시 Operation 이동 검토

클라이언트가 replication switchover를 인지할 수 있는 방법은 두가지다.

  1. operation을 read할 때 SWITCHOVER를 수신할 경우
  2. ZK 이벤트의 캐시 리스트 변경 이벤트 수신 (ZK 이벤트 발생 후 다음 handleIO에서 인지하게 됨)

switchover를 인지하게 된 경우, 구 Master 노드의 readQ,writeQ,inputQ 에 존재하는 모든 Operation들을 구 Slave(신규 Master)로 이동하게 된다.

이때 1번의 경우에는 SWITCHOVER를 수신하게 되면 그 Operation는 서버에 반영이 되지 않았기 때문에 Operation을 구 Slave로 이동하는 것은 아무런 문제가 없다.

하지만 2번의 경우에는 문제가 발생한다.
2번의 경우, readQ에 존재하는 Operation이 이미 서버에 반영이 되었다면, 해당 Operation이 구 Slave로 이동된 후 Operation이 한번 더 요청될 가능성이 존재한다. 따라서 아래와 같은 상황이 발생할 수 있다.

1. incr operation은 반영 완료되었고, readQ에서 읽지 못하여 제거가 되지 않은 상태.
2. ZK 이벤트의 노드 변경으로 인하여 switch over를 인지함.
3. 구 Master의 모든 operation을 신규 Master로 이동
4. 결국 incr operation은 두번 반영됨

2번 상황(ZK 이벤트의 캐시 리스트 변경 이벤트 수신)에 대해 Operation 이동 코드는 아래와 같다.
이에 대해 검토가 필요하다.

https://github.com/naver/arcus-java-client/blob/master/src/main/java/net/spy/memcached/MemcachedConnection.java#L406

bulkService 사용 할 때 NPE 발생 가능성

asyncBulkSet API에서 파라미터를 키 리스트로 사용할 때 NPE가 발생할 수 있다.
API 내부에서 key list가 null인지, empty인지를 확인하고 있지만
key list element 중 null인 키가 존재하는지는 확인하지 않고 있으며
null인 키가 존재할 시 NPE가 발생한다.

maven 배포시 javadoc ERROR, WARNING 발생

jdk 1.8 버전부터 mvn deploy시 java doc 관련 에러가 발생한다.

maven central repository에 배포를 위해 javadoc을 아래와 같이 변경 후 (https://github.com/naver/arcus-java-client/blob/master/pom.xml#L180-L185)

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-javadoc-plugin</artifactId>
                <version>2.7</version>
                <executions>
                    <execution>
                        <id>attach-javadocs</id>
                        <goals>
                            <goal>jar</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <show>public</show>
                    <nohelp>false</nohelp>
                </configuration>
            </plugin>

mvn deploy 혹은 mvn verify를 하면 아래와 같이 javadoc으로 인하여 빌드가 실패된다.

유효하지 않은 javadoc에 대해 수정이 필요하다.

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 9.199s
[INFO] Finished at: Thu Jul 30 16:01:39 KST 2020
[INFO] Final Memory: 19M/312M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-javadoc-plugin:2.7:jar (attach-javadocs) on project arcus-java-client: MavenReportException: Error while creating archive:
[ERROR] Exit code: 1 - /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/AddrUtil.java:21: warning: no @param for s
[ERROR] public static List<InetSocketAddress> getAddresses(String s) {
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/AddrUtil.java:21: warning: no @return
[ERROR] public static List<InetSocketAddress> getAddresses(String s) {
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:73: warning: no description for @param
[ERROR] * @param <T>
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:113: warning: no description for @param
[ERROR] * @param <T>
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:145: warning: no @param for <T>
[ERROR] public abstract <T> Future<Map<String, CollectionOperationStatus>> asyncSetBulk(
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:178: warning: no @param for <T>
[ERROR] public abstract <T> Future<Map<String, CollectionOperationStatus>> asyncBopInsertBulk(
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:211: warning: no description for @param
[ERROR] * @param <T>
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:274: warning: no description for @param
[ERROR] * @param <T>
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:396: error: bad use of '>'
[ERROR] * 	from >= to : in descending order
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:397: error: malformed HTML
[ERROR] * 	from < to  : in ascending order
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:417: warning: no description for @param
[ERROR] * @param <T>
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:434: error: bad use of '>'
[ERROR] * 	from >= to : in descending order
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:435: error: malformed HTML
[ERROR] * 	from < to  : in ascending order
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:438: warning: no description for @param
[ERROR] * @param <T>
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:482: warning: no description for @param
[ERROR] * @param <T>
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:496: warning: no description for @param
[ERROR] * @param <T>
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:523: warning: no description for @param
[ERROR] * @param <T>
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:624: warning: no description for @param
[ERROR] * @param <T>
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:683: warning: no @param for key
[ERROR] public CollectionFuture<Boolean> asyncLopInsert(String key, int index,
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:701: warning: no description for @param
[ERROR] * @param <T>
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:729: warning: no @param for <T>
[ERROR] public <T> CollectionFuture<Boolean> asyncLopInsert(String key, int index,
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:729: warning: no @param for key
[ERROR] public <T> CollectionFuture<Boolean> asyncLopInsert(String key, int index,
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:735: warning: no description for @param
[ERROR] * @param <T>
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:750: warning: no description for @param
[ERROR] * @param elements
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:789: warning: no description for @param
[ERROR] * @param <T>
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:791: warning: no description for @param
[ERROR] * @param elements
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:809: warning: no description for @param
[ERROR] * @param <T>
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:824: warning: no description for @param
[ERROR] * @param <T>
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:919: warning: no @param for <T>
[ERROR] public <T> CollectionFuture<Boolean> asyncBopUpsert(String key, long bkey,
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:958: warning: no @param for <T>
[ERROR] public <T> CollectionFuture<Boolean> asyncBopUpdate(String key, long bkey,
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:984: warning: no @param for <T>
[ERROR] public <T> CollectionFuture<Map<Integer, CollectionOperationStatus>> asyncBopPipedUpdateBulk(
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:1028: warning: no @param for <T>
[ERROR] public <T> CollectionFuture<Boolean> asyncBopInsert(String key,
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:1037: error: bad use of '>'
[ERROR] * 	from >= to : in descending order
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:1038: error: malformed HTML
[ERROR] * 	from < to  : in ascending order
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:1060: error: bad use of '>'
[ERROR] * 	from >= to : in descending order
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:1061: error: malformed HTML
[ERROR] * 	from < to  : in ascending order
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:1075: warning: no @param for <T>
[ERROR] public <T> CollectionFuture<Map<ByteArrayBKey, Element<T>>> asyncBopGet(
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:1130: warning: no @param for <T>
[ERROR] public <T> CollectionFuture<Boolean> asyncBopUpsert(String key,
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:1185: warning: no @param for <T>
[ERROR] public <T> CollectionFuture<Boolean> asyncBopUpdate(String key,
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:1213: warning: no @param for <T>
[ERROR] public <T> CollectionFuture<Map<T, Boolean>> asyncSopPipedExistBulk(
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:1249: warning: no @param for <T>
[ERROR] public <T> CollectionFuture<Map<Integer, CollectionOperationStatus>> asyncBopPipedInsertBulk(
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:1290: warning: no @param for <T>
[ERROR] public <T> CollectionFuture<Map<ByteArrayBKey, Element<T>>> asyncBopGet(
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:1355: warning: no @param for <T>
[ERROR] public abstract <T> Future<Map<String, CollectionOperationStatus>> asyncBopInsertBulk(
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:1399: warning: no @param for <T>
[ERROR] public <T> CollectionGetBulkFuture<Map<String, BTreeGetResult<ByteArrayBKey, T>>> asyncBopGetBulk(
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:1444: warning: no @param for <T>
[ERROR] public <T> CollectionGetBulkFuture<Map<String, BTreeGetResult<Long, T>>> asyncBopGetBulk(
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:1509: warning: no @param for <T>
[ERROR] public <T> CollectionFuture<Map<Integer, Element<T>>> asyncBopGetByPosition(
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:1534: warning: no @param for <T>
[ERROR] public <T> CollectionFuture<Map<Integer, Element<T>>> asyncBopGetByPosition(
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:1582: warning: no @param for <T>
[ERROR] public <T> CollectionFuture<Map<Integer, Element<T>>> asyncBopFindPositionWithGet(
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:1607: warning: no @param for <T>
[ERROR] public <T> CollectionFuture<Map<Integer, Element<T>>> asyncBopFindPositionWithGet(
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:1635: warning: no @param for <E>
[ERROR] public <E> BTreeStoreAndGetFuture<Boolean, E> asyncBopInsertAndGetTrimmed(
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:1664: warning: no @param for <E>
[ERROR] public <E> BTreeStoreAndGetFuture<Boolean, E> asyncBopInsertAndGetTrimmed(
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:1693: warning: no @param for <E>
[ERROR] public <E> BTreeStoreAndGetFuture<Boolean, E> asyncBopUpsertAndGetTrimmed(
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ArcusClientIF.java:1722: warning: no @param for <E>
[ERROR] public <E> BTreeStoreAndGetFuture<Boolean, E> asyncBopUpsertAndGetTrimmed(
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/MemcachedClient.java:387: warning: no description for @param
[ERROR] * @param <T>
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/MemcachedClient.java:424: warning: no description for @param
[ERROR] * @param <T>
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/MemcachedClient.java:441: warning: no description for @param
[ERROR] * @param <T>
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/MemcachedClient.java:458: warning: no description for @param
[ERROR] * @param <T>
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/MemcachedClient.java:527: warning: no description for @param
[ERROR] * @param <T>
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/MemcachedClient.java:546: warning: no description for @param
[ERROR] * @param <T>
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/MemcachedClient.java:627: warning: no description for @param
[ERROR] * @param <T>
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/MemcachedClient.java:698: warning: no description for @param
[ERROR] * @param <T>
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/MemcachedClient.java:770: warning: no description for @param
[ERROR] * @param <T>
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/MemcachedClient.java:822: warning: no description for @param
[ERROR] * @param <T>
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/MemcachedClient.java:873: warning: no description for @param
[ERROR] * @param <T>
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/MemcachedClient.java:923: warning: no description for @param
[ERROR] * @param <T>
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/MemcachedClient.java:962: warning: no description for @param
[ERROR] * @param <T>
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/MemcachedClient.java:1005: warning: no description for @param
[ERROR] * @param <T>
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/MemcachedClient.java:1112: warning: no description for @param
[ERROR] * @param <T>
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/MemcachedClient.java:1140: warning: no description for @param
[ERROR] * @param <T>
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/MemcachedClient.java:1167: warning: no description for @param
[ERROR] * @param <T>
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/MemcachedClient.java:1208: warning: no description for @param
[ERROR] * @param <T>
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/NodeLocator.java:47: warning: no @return
[ERROR] Collection<MemcachedNode> getAll();
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/NodeLocator.java:52: warning: no @return
[ERROR] NodeLocator getReadonlyCopy();
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/NodeLocator.java:58: warning: no @param for toAttach
[ERROR] void update(Collection<MemcachedNode> toAttach, Collection<MemcachedNode> toDelete);
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/NodeLocator.java:58: warning: no @param for toDelete
[ERROR] void update(Collection<MemcachedNode> toAttach, Collection<MemcachedNode> toDelete);
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/DefaultConnectionFactory.java:260: warning: no @return
[ERROR] public int getOpQueueLen() {
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ConnectionFactory.java:50: warning: no @param for sa
[ERROR] MemcachedNode createMemcachedNode(SocketAddress sa,
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ConnectionFactory.java:50: warning: no @param for c
[ERROR] MemcachedNode createMemcachedNode(SocketAddress sa,
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ConnectionFactory.java:50: warning: no @param for bufSize
[ERROR] MemcachedNode createMemcachedNode(SocketAddress sa,
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ConnectionFactory.java:50: warning: no @return
[ERROR] MemcachedNode createMemcachedNode(SocketAddress sa,
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ConnectionFactory.java:56: warning: no @return
[ERROR] BlockingQueue<Operation> createOperationQueue();
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ConnectionFactory.java:62: warning: no @return
[ERROR] BlockingQueue<Operation> createReadOperationQueue();
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ConnectionFactory.java:68: warning: no @return
[ERROR] BlockingQueue<Operation> createWriteOperationQueue();
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ConnectionFactory.java:74: warning: no @return
[ERROR] long getOpQueueMaxBlockTime();
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ConnectionFactory.java:79: warning: no @param for nodes
[ERROR] NodeLocator createLocator(List<MemcachedNode> nodes);
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ConnectionFactory.java:79: warning: no @return
[ERROR] NodeLocator createLocator(List<MemcachedNode> nodes);
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ConnectionFactory.java:85: warning: no @return
[ERROR] OperationFactory getOperationFactory();
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ConnectionFactory.java:90: warning: no @return
[ERROR] long getOperationTimeout();
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ConnectionFactory.java:95: warning: no @return
[ERROR] boolean isDaemon();
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ConnectionFactory.java:104: warning: no @return
[ERROR] boolean useNagleAlgorithm();
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ConnectionFactory.java:112: warning: no @return
[ERROR] Collection<ConnectionObserver> getInitialObservers();
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ConnectionFactory.java:117: warning: no @return
[ERROR] FailureMode getFailureMode();
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ConnectionFactory.java:123: warning: no @return
[ERROR] Transcoder<Object> getDefaultTranscoder();
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ConnectionFactory.java:129: warning: no @return
[ERROR] Transcoder<Object> getDefaultCollectionTranscoder();
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ConnectionFactory.java:134: warning: no @return
[ERROR] boolean shouldOptimize();
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ConnectionFactory.java:139: warning: no @return
[ERROR] int getReadBufSize();
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ConnectionFactory.java:144: warning: no @return
[ERROR] public HashAlgorithm getHashAlg();
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ConnectionFactory.java:149: warning: no @return
[ERROR] long getMaxReconnectDelay();
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ConnectionFactory.java:161: warning: no @return
[ERROR] int getTimeoutExceptionThreshold();
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ConnectionFactory.java:166: warning: no @return
[ERROR] int getTimeoutRatioThreshold();
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ConnectionFactory.java:171: warning: no @return
[ERROR] int getMaxFrontCacheElements();
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ConnectionFactory.java:176: warning: no @return
[ERROR] int getFrontCacheExpireTime();
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ConnectionFactory.java:181: warning: no @return
[ERROR] int getBulkServiceThreadCount();
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ConnectionFactory.java:186: warning: no @return
[ERROR] int getBulkServiceLoopLimit();
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ConnectionFactory.java:191: warning: no @return
[ERROR] long getBulkServiceSingleOpTimeout();
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ConnectionFactory.java:196: warning: no @return
[ERROR] int getDefaultMaxSMGetKeyChunkSize();
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/ConnectionFactory.java:201: warning: no @return
[ERROR] String getFrontCacheName();
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/BinaryConnectionFactory.java:25: warning: no @param for len
[ERROR] public BinaryConnectionFactory(int len, int bufSize) {
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/CacheMonitor.java:31: error: unknown tag: service_code
[ERROR] * in the ZooKeeper node(/arcus/cache_list/<service_code>).
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/CASMutator.java:16: error: unknown tag: Long
[ERROR] * Transcoder<Long> tc = new LongTranscoder();
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/MemcachedNode.java:134: error: unknown tag: q
[ERROR] * True if this node is <q>active.</q>  i.e. is is currently connected
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/MemcachedNode.java:134: error: unknown tag: q
[ERROR] * True if this node is <q>active.</q>  i.e. is is currently connected
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeFindPosition.java:22: error: unknown tag: key
[ERROR] * bop position <key> <bkey> <order>\r\n <position> = 0 or positive integer
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeFindPosition.java:22: error: unknown tag: bkey
[ERROR] * bop position <key> <bkey> <order>\r\n <position> = 0 or positive integer
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeFindPosition.java:22: error: unknown tag: order
[ERROR] * bop position <key> <bkey> <order>\r\n <position> = 0 or positive integer
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeFindPosition.java:22: error: unknown tag: position
[ERROR] * bop position <key> <bkey> <order>\r\n <position> = 0 or positive integer
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeFindPositionWithGet.java:24: error: unknown tag: key
[ERROR] * bop pwg <key> <bkey> <order> [<count>]\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeFindPositionWithGet.java:24: error: unknown tag: bkey
[ERROR] * bop pwg <key> <bkey> <order> [<count>]\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeFindPositionWithGet.java:24: error: unknown tag: order
[ERROR] * bop pwg <key> <bkey> <order> [<count>]\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeFindPositionWithGet.java:24: error: unknown tag: count
[ERROR] * bop pwg <key> <bkey> <order> [<count>]\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeFindPositionWithGet.java:26: error: unknown tag: position
[ERROR] * VALUE <position> <flags> <count> <index>\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeFindPositionWithGet.java:26: error: unknown tag: flags
[ERROR] * VALUE <position> <flags> <count> <index>\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeFindPositionWithGet.java:26: error: unknown tag: count
[ERROR] * VALUE <position> <flags> <count> <index>\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeFindPositionWithGet.java:26: error: unknown tag: index
[ERROR] * VALUE <position> <flags> <count> <index>\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeFindPositionWithGet.java:27: error: unknown tag: bkey
[ERROR] * <bkey> [<eflag>] <bytes> <data>\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeFindPositionWithGet.java:27: error: unknown tag: eflag
[ERROR] * <bkey> [<eflag>] <bytes> <data>\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeFindPositionWithGet.java:27: error: unknown tag: bytes
[ERROR] * <bkey> [<eflag>] <bytes> <data>\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeFindPositionWithGet.java:27: error: unknown tag: data
[ERROR] * <bkey> [<eflag>] <bytes> <data>\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeGetByPosition.java:24: error: unknown tag: key
[ERROR] * bop gbp <key> <order> <position or "position range">\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeGetByPosition.java:24: error: unknown tag: order
[ERROR] * bop gbp <key> <order> <position or "position range">\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeGetByPosition.java:24: error: malformed HTML
[ERROR] * bop gbp <key> <order> <position or "position range">\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeGetByPosition.java:24: error: bad use of '>'
[ERROR] * bop gbp <key> <order> <position or "position range">\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeGetByPosition.java:25: error: unknown tag: flags
[ERROR] * VALUE <flags> <count>\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeGetByPosition.java:25: error: unknown tag: count
[ERROR] * VALUE <flags> <count>\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeGetByPosition.java:26: error: unknown tag: bkey
[ERROR] * <bkey> [<eflag>] <bytes> <data>\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeGetByPosition.java:26: error: unknown tag: eflag
[ERROR] * <bkey> [<eflag>] <bytes> <data>\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeGetByPosition.java:26: error: unknown tag: bytes
[ERROR] * <bkey> [<eflag>] <bytes> <data>\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeGetByPosition.java:26: error: unknown tag: data
[ERROR] * <bkey> [<eflag>] <bytes> <data>\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeStoreAndGet.java:23: error: unknown tag: key
[ERROR] *     - bop insert <key> <bkey> [<eflag>] <bytes> [create <attributes>] getrim\r\n<data>\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeStoreAndGet.java:23: error: unknown tag: bkey
[ERROR] *     - bop insert <key> <bkey> [<eflag>] <bytes> [create <attributes>] getrim\r\n<data>\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeStoreAndGet.java:23: error: unknown tag: eflag
[ERROR] *     - bop insert <key> <bkey> [<eflag>] <bytes> [create <attributes>] getrim\r\n<data>\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeStoreAndGet.java:23: error: unknown tag: bytes
[ERROR] *     - bop insert <key> <bkey> [<eflag>] <bytes> [create <attributes>] getrim\r\n<data>\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeStoreAndGet.java:23: error: unknown tag: attributes
[ERROR] *     - bop insert <key> <bkey> [<eflag>] <bytes> [create <attributes>] getrim\r\n<data>\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeStoreAndGet.java:23: error: unknown tag: data
[ERROR] *     - bop insert <key> <bkey> [<eflag>] <bytes> [create <attributes>] getrim\r\n<data>\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeStoreAndGet.java:24: error: unknown tag: key
[ERROR] *     - bop upsert <key> <bkey> [<eflag>] <bytes> [create <attributes>] getrim\r\n<data>\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeStoreAndGet.java:24: error: unknown tag: bkey
[ERROR] *     - bop upsert <key> <bkey> [<eflag>] <bytes> [create <attributes>] getrim\r\n<data>\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeStoreAndGet.java:24: error: unknown tag: eflag
[ERROR] *     - bop upsert <key> <bkey> [<eflag>] <bytes> [create <attributes>] getrim\r\n<data>\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeStoreAndGet.java:24: error: unknown tag: bytes
[ERROR] *     - bop upsert <key> <bkey> [<eflag>] <bytes> [create <attributes>] getrim\r\n<data>\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeStoreAndGet.java:24: error: unknown tag: attributes
[ERROR] *     - bop upsert <key> <bkey> [<eflag>] <bytes> [create <attributes>] getrim\r\n<data>\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeStoreAndGet.java:24: error: unknown tag: data
[ERROR] *     - bop upsert <key> <bkey> [<eflag>] <bytes> [create <attributes>] getrim\r\n<data>\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeStoreAndGet.java:25: error: unknown tag: flags
[ERROR] *     VALUE <flags> <count>\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeStoreAndGet.java:25: error: unknown tag: count
[ERROR] *     VALUE <flags> <count>\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeStoreAndGet.java:26: error: unknown tag: bkey
[ERROR] *     <bkey> [<eflag>] <bytes> <data>\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeStoreAndGet.java:26: error: unknown tag: eflag
[ERROR] *     <bkey> [<eflag>] <bytes> <data>\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeStoreAndGet.java:26: error: unknown tag: bytes
[ERROR] *     <bkey> [<eflag>] <bytes> <data>\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/collection/BTreeStoreAndGet.java:26: error: unknown tag: data
[ERROR] *     <bkey> [<eflag>] <bytes> <data>\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/protocol/ascii/BTreeSortMergeGetOperationImpl.java:80: error: unknown tag: flag
[ERROR] * VALUE <flag> <count>\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/protocol/ascii/BTreeSortMergeGetOperationImpl.java:80: error: unknown tag: count
[ERROR] * VALUE <flag> <count>\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/protocol/ascii/CollectionCountOperationImpl.java:64: error: unknown tag: flag
[ERROR] * VALUE <flag> <count>\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/protocol/ascii/CollectionCountOperationImpl.java:64: error: unknown tag: count
[ERROR] * VALUE <flag> <count>\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/protocol/ascii/CollectionGetOperationImpl.java:87: error: unknown tag: flag
[ERROR] * VALUE <flag> <count>\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/protocol/ascii/CollectionGetOperationImpl.java:87: error: unknown tag: count
[ERROR] * VALUE <flag> <count>\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/protocol/ascii/CollectionMutateOperationImpl.java:65: error: unknown tag: result
[ERROR] * <result value>\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/protocol/ascii/ExtendedBTreeGetOperationImpl.java:88: error: unknown tag: flag
[ERROR] * VALUE <flag> <count>\r\n
[ERROR] ^
[ERROR] /home/jam2in/workspace/arcus-java-client/src/main/java/net/spy/memcached/protocol/ascii/ExtendedBTreeGetOperationImpl.java:88: error: unknown tag: count
[ERROR] * VALUE <flag> <count>\r\n
[ERROR] ^
[ERROR]
[ERROR] Command line was: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre/../bin/javadoc @options @packages
[ERROR]
[ERROR] Refer to the generated Javadoc files in '/home/jam2in/workspace/arcus-java-client/target/apidocs' dir.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

Command pipeline response 처리 이슈.

현재 command pipeline response 처리 상의 아래 이슈가 있다.

  • PIPE_ERROR string에 대한 handling이 없다.
    • PIPE_ERROR string을 받으면, 현재 operation을 완료하여야 한다.
  • PIPE 연산으로 수행할 연산이 1개인 경우에 제대로 처리하여야 한다.

Acrus znode path 관련 정리.

Arcus znode path 변수명 변경

  • CACHE_LIST_PATH => ARCUS_BASE_CACHE_LIST_ZPATH
  • CLIENT_INFO_PATH => ARCUS_BASE_CLIENT_LIST_ZPATH
  • 기타) 향후 replication 기능 가정하면,
    • ARCUS_REPL_CACHE_LIST_ZPATH
    • ARCUS_REPL_CLIENT_LIST_ZPATH

ZK session timeout 상수명 변경

  • SESSEION_TIMEOUT => ZK_SESSION_TIMEOUT

CacheMonitor 클래스에 cacheListZPath 추가하여 관리

  • CacheMonitor 생성 시에 cacheListZPath 전달 필요

multi-key delete 기능

하나의 delete 명령으로 multiple keys를 삭제하는 기능입니다.

  • 요구 사항
    • 응용에서 다수의 키를 삭제하는 경우 현재는 loop 통해 개별 키를 삭제하고 있다.
    • 이 경우, 전체 키를 삭제하는 latency가 길어지는 단점이 있다.
  • 해결 방안
    • 한번의 delete 명령으로 다수의 키를 삭제하도록 한다.
    • 하나의 작업 스레드가 다수 커넥션들의 요청들을 큰 지연없이 처리하여야 한다.
    • 이를 위해 O(small N) 제약을 준수해야 하며, 대략 100개 key들을 한번에 delete하도록 한다.
    • Cache Server, Java/C Client 모두 수정이 필요하다.

Arcus java client 시작 시 ZK connect timeout 상태이면, "ConnectionLossException: KeeperErrorCode = ConnectionLoss for ..." 발생

Arcus java client 시작 시 Arcus Admin인 ZK connect timeout 상태이면,

  • "ConnectionLossException: KeeperErrorCode = ConnectionLoss for ..." exception 발생하고,
  • "Unexpected exception. contact to Arcus administrator" 메세지가 출력된다.

즉, 사용자는 현재 exception의 의미를 정확히 이해하기 어렵다...

위의 처리 방식 대신에
"Admin에 연결 요청이 timeout 되었음"을 정의하는 exception을 명시적으로
던지도록 하는 것이 사용자가 현 상황을 이해하고 적절히 대응하는 데 도움이 된다.

Javadoc의 특수 기호 표현 방식 개선

javadoc에서 generic 기호(<>)를 사용하면 HTML tag로 인식하는 문제가 있어,
< 문자는 &lt; 로 대체하고, > 문자는 &gt;로 대체하여 사용한다.

예를 들어 javadoc에서 List<String> list = Collection.singletonList("list") 라는 코드를 javadoc에 삽입하려고 하면, 아래와 같이 작성되어야 한다.

/**
 * <pre>
 * List&lt;String&gt; list = Collection.singletonList(&quot;list&quot;)
 * </pre>
 */

하지만, @code를 사용하면 특수 기호를 그대로 javadoc에 표현할 수 있다.

/**
 * <pre>{@code
 * List<String> list = Collection.singletonList("list")
 * }</pre>
 */

현재 arcus-java-client에서는 특수기호를 표현하기 위해 ampersand('&')로 시작하는 문자 코드를 사용하고 있으며, @code 를 사용하여 javadoc을 가독성있게 바꿀 수 있도록 수정한다.

참고 링크: https://reflectoring.io/howto-format-code-snippets-in-javadoc/

addAllOpToInputQ()에서 inputQ에 연산 삽입의 실패 처리 필요

addAllOpToInputQ() 메소드에서 inputQ에 연산을 삽입하는 코드는 아래와 같다.

  public int addAllOpToInputQ(BlockingQueue<Operation> allOp) {
    int movedOpCount = 0;
    for (Operation op : allOp) {
      if (inputQueue.remainingCapacity() == 0) {
        op.cancel("by moving operations");
      } else {
        op.setHandlingNode(this);
        if ((op.getState() == OperationState.WRITE_QUEUED ||
             op.getState() == OperationState.WRITING) && op.getBuffer() != null) {
          op.getBuffer().reset(); // buffer offset reset
        } else {
          op.initialize(); // write completed or not yet initialized
          op.resetState(); // reset operation state
        }
        op.setMoved(true);
        movedOpCount++;
        inputQueue.offer(op);
      }
    }
    addOpCount += movedOpCount;
    return movedOpCount;
  }

문제의 설명은 다음과 같다.

  • inputQ에 remaining space가 남아 있더라도 inputQ에 실제로 연산을 삽입할 시점에는 full 상태일 수 있다.
    • 이유는 work thread에 의한 다른 연산이 삽입될 수 있기 때문이다.

이를 해결하는 구현은 다음과 같다.

  • inputQ에 remaining space가 남아 있는 지를 검사할 필요가 없다.
  • inputQ에 offer하여 실패가 리턴되면 해당 연산을 cancel 하면 된다.

bulkServiceLoopLimit 기본값에 대한 검토

bulkServiceLoopLimit는 현재 default 값이 1로 설정되어 있다.
이렇게 설정되어 있을 경우 매번 operation이 전달된것을 기다리기 때문에
bulk를 사용하는 것과 사용하지 않는 것의 차이가 없다.
기본값을 변경해가며 성능을 측정한 후 기본값에 적절한 값으로 수정한다.

Operation Timeout 로그에 TimeUnit 추가

본 이슈는 jam2in arcus-java-client Operation Timeout 로그에 TimeUnit 추가 에서 옮겨왔습니다.


operation timeout 발생 시 아래와 같이 로그가 출력되고 있습니다.

net.spy.memcached.internal.CheckedOperationTimeoutException: Timed out waiting for operation. >3 - failing node: /10.144.146.17:11211 [READING] [#
Tops=406766 #iq=1 #Wops=0 #Rops=1 #CT=1 #TR=-1]

위에서 3이라는 timeout 값이 어떤 단위를 사용하고 있는지 전혀 알 수 없는 상태입니다.
로그에서 TimeUnit 값을 함께 출력하는 것이 분석에 도움이 될 것 같습니다.

BulkGet 요청 처리 재검증

bulk get 수행 방식이 현재 최적인지 확인하고, 개선 부분을 도출한다.
특히, 아래 경우들을 고려한다.

  • inactive node로 mapping되는 key들이 존재할 경우
  • 해당 연산이 하나라도 오류가 발생할 경우
  • 해당 연산이 하나라도 timeout 발생할 경우

추가적으로, local cache 활용도 정상적인지 확인한다.

Operation 이동에 대한 addAllOpToInputQ 동작 개선

Replication switchover 또는 failure가 발생 후
master에서 slave의 inputQ(TCPMemcachedNodeImpl)로 operation을 이동할 때,
slave의 inputQ의 capacity를 체크하고, full 상태이면 operation을 cancel하도록 처리하고 있다.

public void addAllOpToInputQ(BlockingQueue<Operation> allOp) {
  for (Operation op : allOp) {
    if (inputQueue.remainingCapacity() == 0) {
      op.cancel("by moving operations");
    } else {
        op.setHandlingNode(this);
        if ((op.getState() == OperationState.WRITE_QUEUED ||
             op.getState() == OperationState.WRITING) && op.getBuffer() != null) {
          op.getBuffer().reset(); // buffer offset reset
        } else {
          op.initialize(); // write completed or not yet initialized
          op.resetState(); // reset operation state
        }
        op.setMoved(true);
      }
      addOpCount++;
      inputQueue.offer(op);
    }
}

slave의 inputQ가 가득찬 상태에서도 이러한 operation들을 cancel 처리하지 않도록 하는 방법이 필요하다.

maven checkstyle java version 1.8 이상 요구

maven checkstyle 은 java version 1.8 이상을 요구한다. 그 이하 환경에서 mvn install 하면 아래와 같은 에러 메세지가 발생한다. arcus-java-client를 사용하고 있는 곳에서의 java 버전을 확인하고, 어떤 버전까지 지원할지 결정 필요.

mvn clean install -DskipTests=true
[INFO] Scanning for projects...
[INFO]
[INFO] ---------------< com.navercorp.arcus:arcus-java-client >----------------
[INFO] Building Arcus Java Client 1.12.0
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ arcus-java-client ---
[INFO]
[INFO] --- maven-checkstyle-plugin:3.1.1:check (validate) @ arcus-java-client ---
[WARNING] Error injecting: org.apache.maven.plugins.checkstyle.CheckstyleViolationCheckMojo
java.lang.TypeNotPresentException: Type org.apache.maven.plugins.checkstyle.CheckstyleViolationCheckMojo not present
    at org.eclipse.sisu.space.URLClassSpace.loadClass (URLClassSpace.java:147)
    at org.eclipse.sisu.space.NamedClass.load (NamedClass.java:46)
    at org.eclipse.sisu.space.AbstractDeferredClass.get (AbstractDeferredClass.java:48)
    at com.google.inject.internal.ProviderInternalFactory.provision (ProviderInternalFactory.java:85)
    at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision (InternalFactoryToInitializableAdapter.java:57)
    at com.google.inject.internal.ProviderInternalFactory$1.call (ProviderInternalFactory.java:66)
    at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision (ProvisionListenerStackCallback.java:112)
    at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision (ProvisionListenerStackCallback.java:127)
    at com.google.inject.internal.ProvisionListenerStackCallback.provision (ProvisionListenerStackCallback.java:66)
    at com.google.inject.internal.ProviderInternalFactory.circularGet (ProviderInternalFactory.java:61)
    at com.google.inject.internal.InternalFactoryToInitializableAdapter.get (InternalFactoryToInitializableAdapter.java:47)
    at com.google.inject.internal.InjectorImpl$1.get (InjectorImpl.java:1050)
    at org.eclipse.sisu.inject.Guice4$1.get (Guice4.java:162)
    at org.eclipse.sisu.inject.LazyBeanEntry.getValue (LazyBeanEntry.java:81)
    at org.eclipse.sisu.plexus.LazyPlexusBean.getValue (LazyPlexusBean.java:51)
    at org.codehaus.plexus.DefaultPlexusContainer.lookup (DefaultPlexusContainer.java:263)
    at org.codehaus.plexus.DefaultPlexusContainer.lookup (DefaultPlexusContainer.java:255)
    at org.apache.maven.plugin.internal.DefaultMavenPluginManager.getConfiguredMojo (DefaultMavenPluginManager.java:520)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:124)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:607)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)
Caused by: java.lang.UnsupportedClassVersionError: org/apache/maven/plugins/checkstyle/CheckstyleViolationCheckMojo : Unsupported major.minor version 52.0
    at java.lang.ClassLoader.defineClass1 (Native Method)
    at java.lang.ClassLoader.defineClass (ClassLoader.java:808)
    at java.security.SecureClassLoader.defineClass (SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass (URLClassLoader.java:443)
    at java.net.URLClassLoader.access$100 (URLClassLoader.java:65)
    at java.net.URLClassLoader$1.run (URLClassLoader.java:355)
    at java.net.URLClassLoader$1.run (URLClassLoader.java:349)
    at java.security.AccessController.doPrivileged (Native Method)
    at java.net.URLClassLoader.findClass (URLClassLoader.java:348)
    at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClassFromSelf (ClassRealm.java:401)
    at org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass (SelfFirstStrategy.java:42)
    at org.codehaus.plexus.classworlds.realm.ClassRealm.unsynchronizedLoadClass (ClassRealm.java:271)
    at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass (ClassRealm.java:247)
    at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass (ClassRealm.java:239)
    at org.eclipse.sisu.space.URLClassSpace.loadClass (URLClassSpace.java:139)
    at org.eclipse.sisu.space.NamedClass.load (NamedClass.java:46)
    at org.eclipse.sisu.space.AbstractDeferredClass.get (AbstractDeferredClass.java:48)
    at com.google.inject.internal.ProviderInternalFactory.provision (ProviderInternalFactory.java:85)
    at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision (InternalFactoryToInitializableAdapter.java:57)
    at com.google.inject.internal.ProviderInternalFactory$1.call (ProviderInternalFactory.java:66)
    at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision (ProvisionListenerStackCallback.java:112)
    at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision (ProvisionListenerStackCallback.java:127)
    at com.google.inject.internal.ProvisionListenerStackCallback.provision (ProvisionListenerStackCallback.java:66)
    at com.google.inject.internal.ProviderInternalFactory.circularGet (ProviderInternalFactory.java:61)
    at com.google.inject.internal.InternalFactoryToInitializableAdapter.get (InternalFactoryToInitializableAdapter.java:47)
    at com.google.inject.internal.InjectorImpl$1.get (InjectorImpl.java:1050)
    at org.eclipse.sisu.inject.Guice4$1.get (Guice4.java:162)
    at org.eclipse.sisu.inject.LazyBeanEntry.getValue (LazyBeanEntry.java:81)
    at org.eclipse.sisu.plexus.LazyPlexusBean.getValue (LazyPlexusBean.java:51)
    at org.codehaus.plexus.DefaultPlexusContainer.lookup (DefaultPlexusContainer.java:263)
    at org.codehaus.plexus.DefaultPlexusContainer.lookup (DefaultPlexusContainer.java:255)
    at org.apache.maven.plugin.internal.DefaultMavenPluginManager.getConfiguredMojo (DefaultMavenPluginManager.java:520)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:124)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:607)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  3.001 s
[INFO] Finished at: 2020-04-29T18:51:39+09:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-checkstyle-plugin:3.1.1:check (validate) on project arcus-java-client: Execution validate of goal org.apache.maven.plugins:maven-checkstyle-plugin:3.1.1:check failed: Unable to load the mojo 'check' in the plugin 'org.apache.maven.plugins:maven-checkstyle-plugin:3.1.1' due to an API incompatibility: org.codehaus.plexus.component.repository.exception.ComponentLookupException: org/apache/maven/plugins/checkstyle/CheckstyleViolationCheckMojo : Unsupported major.minor version 52.0
[ERROR] -----------------------------------------------------
[ERROR] realm =    plugin>org.apache.maven.plugins:maven-checkstyle-plugin:3.1.1
[ERROR] strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy
[ERROR] urls[0] = file:/home/suhwan/.m2/repository/org/apache/maven/plugins/maven-checkstyle-plugin/3.1.1/maven-checkstyle-plugin-3.1.1.jar
[ERROR] urls[1] = file:/home/suhwan/.m2/repository/org/sonatype/aether/aether-util/1.7/aether-util-1.7.jar
[ERROR] urls[2] = file:/home/suhwan/.m2/repository/org/sonatype/sisu/sisu-inject-bean/1.4.2/sisu-inject-bean-1.4.2.jar
[ERROR] urls[3] = file:/home/suhwan/.m2/repository/org/sonatype/sisu/sisu-guice/2.1.7/sisu-guice-2.1.7-noaop.jar
[ERROR] urls[4] = file:/home/suhwan/.m2/repository/org/sonatype/plexus/plexus-sec-dispatcher/1.3/plexus-sec-dispatcher-1.3.jar
[ERROR] urls[5] = file:/home/suhwan/.m2/repository/org/sonatype/plexus/plexus-cipher/1.4/plexus-cipher-1.4.jar
[ERROR] urls[6] = file:/home/suhwan/.m2/repository/org/apache/maven/reporting/maven-reporting-api/3.0/maven-reporting-api-3.0.jar
[ERROR] urls[7] = file:/home/suhwan/.m2/repository/org/apache/maven/reporting/maven-reporting-impl/2.3/maven-reporting-impl-2.3.jar
[ERROR] urls[8] = file:/home/suhwan/.m2/repository/org/apache/maven/shared/maven-shared-utils/0.6/maven-shared-utils-0.6.jar
[ERROR] urls[9] = file:/home/suhwan/.m2/repository/com/google/code/findbugs/jsr305/2.0.1/jsr305-2.0.1.jar
[ERROR] urls[10] = file:/home/suhwan/.m2/repository/org/apache/maven/doxia/doxia-core/1.2/doxia-core-1.2.jar
[ERROR] urls[11] = file:/home/suhwan/.m2/repository/xerces/xercesImpl/2.9.1/xercesImpl-2.9.1.jar
[ERROR] urls[12] = file:/home/suhwan/.m2/repository/xml-apis/xml-apis/1.3.04/xml-apis-1.3.04.jar
[ERROR] urls[13] = file:/home/suhwan/.m2/repository/org/apache/httpcomponents/httpclient/4.0.2/httpclient-4.0.2.jar
[ERROR] urls[14] = file:/home/suhwan/.m2/repository/org/apache/httpcomponents/httpcore/4.0.1/httpcore-4.0.1.jar
[ERROR] urls[15] = file:/home/suhwan/.m2/repository/commons-codec/commons-codec/1.3/commons-codec-1.3.jar
[ERROR] urls[16] = file:/home/suhwan/.m2/repository/commons-validator/commons-validator/1.3.1/commons-validator-1.3.1.jar
[ERROR] urls[17] = file:/home/suhwan/.m2/repository/commons-digester/commons-digester/1.6/commons-digester-1.6.jar
[ERROR] urls[18] = file:/home/suhwan/.m2/repository/commons-logging/commons-logging/1.0.4/commons-logging-1.0.4.jar
[ERROR] urls[19] = file:/home/suhwan/.m2/repository/org/apache/maven/doxia/doxia-sink-api/1.4/doxia-sink-api-1.4.jar
[ERROR] urls[20] = file:/home/suhwan/.m2/repository/org/apache/maven/doxia/doxia-logging-api/1.4/doxia-logging-api-1.4.jar
[ERROR] urls[21] = file:/home/suhwan/.m2/repository/org/apache/maven/doxia/doxia-decoration-model/1.4/doxia-decoration-model-1.4.jar
[ERROR] urls[22] = file:/home/suhwan/.m2/repository/org/apache/maven/doxia/doxia-site-renderer/1.4/doxia-site-renderer-1.4.jar
[ERROR] urls[23] = file:/home/suhwan/.m2/repository/org/apache/maven/doxia/doxia-module-xhtml/1.4/doxia-module-xhtml-1.4.jar
[ERROR] urls[24] = file:/home/suhwan/.m2/repository/org/apache/maven/doxia/doxia-module-fml/1.4/doxia-module-fml-1.4.jar
[ERROR] urls[25] = file:/home/suhwan/.m2/repository/org/codehaus/plexus/plexus-i18n/1.0-beta-7/plexus-i18n-1.0-beta-7.jar
[ERROR] urls[26] = file:/home/suhwan/.m2/repository/org/apache/velocity/velocity-tools/2.0/velocity-tools-2.0.jar
[ERROR] urls[27] = file:/home/suhwan/.m2/repository/commons-chain/commons-chain/1.1/commons-chain-1.1.jar
[ERROR] urls[28] = file:/home/suhwan/.m2/repository/dom4j/dom4j/1.1/dom4j-1.1.jar
[ERROR] urls[29] = file:/home/suhwan/.m2/repository/oro/oro/2.0.8/oro-2.0.8.jar
[ERROR] urls[30] = file:/home/suhwan/.m2/repository/sslext/sslext/1.2-0/sslext-1.2-0.jar
[ERROR] urls[31] = file:/home/suhwan/.m2/repository/org/apache/struts/struts-core/1.3.8/struts-core-1.3.8.jar
[ERROR] urls[32] = file:/home/suhwan/.m2/repository/org/apache/struts/struts-taglib/1.3.8/struts-taglib-1.3.8.jar
[ERROR] urls[33] = file:/home/suhwan/.m2/repository/org/apache/struts/struts-tiles/1.3.8/struts-tiles-1.3.8.jar
[ERROR] urls[34] = file:/home/suhwan/.m2/repository/org/apache/maven/doxia/doxia-integration-tools/1.6/doxia-integration-tools-1.6.jar
[ERROR] urls[35] = file:/home/suhwan/.m2/repository/commons-io/commons-io/1.4/commons-io-1.4.jar
[ERROR] urls[36] = file:/home/suhwan/.m2/repository/junit/junit/3.8.1/junit-3.8.1.jar
[ERROR] urls[37] = file:/home/suhwan/.m2/repository/org/codehaus/plexus/plexus-component-annotations/2.0.0/plexus-component-annotations-2.0.0.jar
[ERROR] urls[38] = file:/home/suhwan/.m2/repository/org/codehaus/plexus/plexus-resources/1.1.0/plexus-resources-1.1.0.jar
[ERROR] urls[39] = file:/home/suhwan/.m2/repository/org/codehaus/plexus/plexus-utils/3.3.0/plexus-utils-3.3.0.jar
[ERROR] urls[40] = file:/home/suhwan/.m2/repository/org/codehaus/plexus/plexus-interpolation/1.26/plexus-interpolation-1.26.jar
[ERROR] urls[41] = file:/home/suhwan/.m2/repository/org/codehaus/plexus/plexus-velocity/1.2/plexus-velocity-1.2.jar
[ERROR] urls[42] = file:/home/suhwan/.m2/repository/com/puppycrawl/tools/checkstyle/8.29/checkstyle-8.29.jar
[ERROR] urls[43] = file:/home/suhwan/.m2/repository/info/picocli/picocli/4.1.4/picocli-4.1.4.jar
[ERROR] urls[44] = file:/home/suhwan/.m2/repository/antlr/antlr/2.7.7/antlr-2.7.7.jar
[ERROR] urls[45] = file:/home/suhwan/.m2/repository/org/antlr/antlr4-runtime/4.8-1/antlr4-runtime-4.8-1.jar
[ERROR] urls[46] = file:/home/suhwan/.m2/repository/commons-beanutils/commons-beanutils/1.9.4/commons-beanutils-1.9.4.jar
[ERROR] urls[47] = file:/home/suhwan/.m2/repository/com/google/guava/guava/28.2-jre/guava-28.2-jre.jar
[ERROR] urls[48] = file:/home/suhwan/.m2/repository/com/google/guava/failureaccess/1.0.1/failureaccess-1.0.1.jar
[ERROR] urls[49] = file:/home/suhwan/.m2/repository/com/google/guava/listenablefuture/9999.0-empty-to-avoid-conflict-with-guava/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
[ERROR] urls[50] = file:/home/suhwan/.m2/repository/org/checkerframework/checker-qual/2.10.0/checker-qual-2.10.0.jar
[ERROR] urls[51] = file:/home/suhwan/.m2/repository/com/google/errorprone/error_prone_annotations/2.3.4/error_prone_annotations-2.3.4.jar
[ERROR] urls[52] = file:/home/suhwan/.m2/repository/com/google/j2objc/j2objc-annotations/1.3/j2objc-annotations-1.3.jar
[ERROR] urls[53] = file:/home/suhwan/.m2/repository/net/sf/saxon/Saxon-HE/9.9.1-6/Saxon-HE-9.9.1-6.jar
[ERROR] urls[54] = file:/home/suhwan/.m2/repository/org/apache/velocity/velocity/1.7/velocity-1.7.jar
[ERROR] urls[55] = file:/home/suhwan/.m2/repository/commons-lang/commons-lang/2.4/commons-lang-2.4.jar
[ERROR] urls[56] = file:/home/suhwan/.m2/repository/commons-collections/commons-collections/3.2.2/commons-collections-3.2.2.jar
[ERROR] Number of foreign imports: 1
[ERROR] import: Entry[import  from realm ClassRealm[project>com.navercorp.arcus:arcus-java-client:1.12.0, parent: ClassRealm[maven.api, parent: null]]]
[ERROR]
[ERROR] -----------------------------------------------------
[ERROR]
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginContainerException

Bulk Service 동작 최적화 검토.

Bulk Service에서 awaitProcessResult() 동작 방식은 timed wait 방식을 사용하고 있다.
이 방식이 bulk service에 적합하지 않다고 본다.

예를 들어, 동시에 10개 연산을 수행할 수 있다고 가정한다.

  • 처음 10개 연산을 요청한다.
  • awaitProcessResult()는 첫째 연산부터 차례로 그 연산이 완료되기를 기다린다.
    • 완료되기를 기다리기 위해. future.get() 메소드를 이용한다.
    • 첫째 연산에서 timeout 발생한다면, 그 기간 동안 다음 연산을 요청할 수 없다.
    • timeout 발생하는 연산에 비례하여 전체 응답 시간이 길어진다.
  • 이 방식 대신에 future.isDone() 메소드로 대기 없이
    해당 연산이 완료되었는 지를 검사하는 방식이 나을 것 같다.
    • 이 경우, 10개 중에 어느 하나라도 완료되면, 다음 연산을 요청할 수 있다.
    • 만약, 10개 모두 완료되지 않았다면, 첫째 연산에 대해 future.get()으로 대기해야 한다.

검토 바랍니다.

Operation clone에서 누락된 Operation에 대해 검토

handleIO에서 operation들을 redistribute할 때, 기존 operation을 clone하여 다시 inputQ에 삽입한다.
operation의 clone은 BaseOperationFactory 클래스가 담당한다.

아래 코드를 참고하면 if 조건으로 operation type에 따라 clone 하고 있다.

https://github.com/naver/arcus-java-client/blob/master/src/main/java/net/spy/memcached/ops/BaseOperationFactory.java#L38-L103

  • 여기서 CollectionUpsertOperation과 같은 누락된 Operation이 없는지 확인하고, 누락된 Operation 들도 clone이 될 수 있도록 추가한다.
  • BaseOperationFactory 클래스는 Operation 클래스가 추가될 때마다 clone 코드를 수정해야 하는 문제가 있다. 만약 다른 개발자가 Operation을 추가한 후, clone에 추가하지 않았다면 현재 이슈와 똑같은 상황이 발생될 수 있다. BaseOperationFactory 클래스를 수정하지 않고 operation을 clone할 수 있는 다른 방법이 필요하다.

불필요한 package 경로 제거

ArcusClient.java 에 보면 아래와 같이 CacheManger 를 생성할 때 net.spy.memcached 와 같이 package 경로를 사용하는 부분이 있습니다.

    private static ArcusClientPool createArcusClient(String hostPorts, String serviceCode,
            ConnectionFactoryBuilder cfb, int poolSize, int waitTimeForConnect) {
...
        net.spy.memcached.CacheManager exe = new net.spy.memcached.CacheManager(
                hostPorts, serviceCode, cfb, latch, poolSize,
                waitTimeForConnect);

같은 package 내에 있고 다른 위치에서는 아래와 같이 package 경로를 사용하지 않고 있습니다.

    static final int MAX_SMGET_COUNT = 1000; // server configuration is 2000.

    private CacheManager cacheManager;

기존 ArcusClient.java 의 위치가 다른 곳에 있었던 흔적이 아닐까 생각됩니다.
특별한 이유가 없다면 제거해도 무방할 것 같습니다.

debug log 출력 방식 일치

본 이슈는 jam2in arcus-java-client debug log 출력 방식 일치 에서 옮겨왔습니다.

java client 전체적으로 debug log 를 출력할 때 아래와 같이 두 가지 방식을 혼용하고 있다.
한 가지 방식으로 통일한다.

        getLogger().debug("Got line %s", line);
        if (getLogger().isDebugEnabled()) {
            getLogger().debug("Got line %s", line);
        }

특정 key를 담당하는 cache node 찾을 시에 failure mode 고려 필요

Key-to-node 찾을 시에 failure mode를 고려하고 있는 함수는 아래와 같다.

  • MemcachedConnection의 addOperation()

Key-to-node 찾을 시에 설정된 failure mode를 고려하지 않고, redistribute 모드로 동작하는 경우가 있다.

  • MemcachedConnection의 findNodeByKey()
  • asyncGetBulk()

위의 두 경우에서도 failure mode에 따라 특정 key를 담당하는 cache node를 찾도록 변경되어야 한다.

연산에 대한 수행 오류 발생 시 응용에서의 권장 처리 방법 가이드 작성

응용에서 연산들을 생성하였을 때 여러 이유로 해당 연산이 수행되지 못하고, 오류가 발생할 수 있다.
이 때 응용에서 상황에 따라 어떻게 처리를 해야하는지에 대한 가이드를 docs에 작성해 놓을 필요가 있다.
이러한 오류 처리는 연산의 타입(키밸류, 콜렉션)이나 단일 연산인지 bulk 인지에 따라 다르게 가이드 될 수 있다.

ketama hash collision 시의 owner 결정

여러 노드의 160 hash point 중에 서로 collistion 발생 시,

  • node의 name string 대소 비교하여,
  • 더 작은 name string을 가진 노드가 그 hash point에 대한 ownership을 가지게 처리

key-value 연산들의 반환 타입 변경 검토

현재 key-value 연산들의 반환타입은 Future으로,
Boolean값으로 단순 연산의 성공/실패 여부만을 확인할 수 있다.
이는 collection 연산들에서 오류 원인을 메시지로 주는 것과 차이가 있다.
이에 반환 타입을 변경하는 것을 검토한다.

CollectionStoreOperation 정리 검토.

collection에 대한 insert/upsert/update 연산을 CollectionStoreOperation 객체로
merge하는 게 좋을 지를 검토하고, merge하는 게 좋다면 그렇게 구현합시다.

key-to-node mapping에 참여하지 않는 key의 postfix 기능 개발.

key의 postfix string이란 key-to-node mapping을 위한 consistent hashing에 사용되지 않는 string을 의미한다.
이를 통해, 특정 key들이 동일 cache node로 mapping될 수 있도록 한다.

예를 들면, 아래의 두 키가 같은 cache node로 mapping 되게 한다. 아래에서 postfix delimiter는 ^ 문자를 가정한 것이다.

  • <"prefix1:subkey1^postfix1" - "value1">
  • <"prefix1:subkey1^postfix2" - "value2">

Operation timeout 시의 출력 정보 추가 - # of added operations.

ARCUS java client에서 operation timeout 난 경우, 현재 아래의 정보를 남깁니다.

  • cache node IP:port
  • operation type(READING or WRITING)
  • iq(input queue size)
  • Wops(write queue size)
  • Rops(read queue size)
  • CT(continuous timeouts)

Continuous timeout 발생 상황에서
과도한 operation이 계속 들어온 것인지 또는 operation 처리가 늦은 것인 지의
판별을 위해 # of added operations 정보를 출력해 두면 유용할 것 같습니다.

ARCUS client의 초당 처리량을 정확히 측정하여 logging하면 좋겠지만,
이 방식은 timer를 돌려야 하는 부담이 있습니다.
따라서, # of added operations 만을 logging하는 게 추가 비용을 들이지 않으면서
대략적인 초당 처리량을 알 수 있는 방법이라 생각합니다.

특정 key를 담당하는 cache node 찾는 로직 최적화 필요

KetamaNodeLocator 그리고 ArcusKetamaNodeLocator에서
특정 key string에 대한 memcached node를 찾는 로직을 보면, 아래와 같이 tailMap을 사용하여 찾고 있는 데요.. 매우 비효율적이라 생각됩니다..

ketamaNodes.tailMap(hash)

TreeMap에서 주어진 key hash 값 이상의 node hash 값을 가진 entry 하나 만을 찾는 기능으로
floorKey API가 있습니다.
Java version에 따라 지원 여부를 확인해 보고 floorKey API 사용하도록 수정하는 게 좋을 것 같습니다.

VersionOperation 처리 검토

VersionOperation이 취소되거나 실패가 발생한 후, 도중 BTree GetBulk와 같은 Operation이 추가될 때, 이 Operation이 Version Operation보다 inputQ 앞단에 위치하게되어, 서버가 잘못된 프로토콜의 Operation을 수신받을 수 있음.

  1. Application 시작
  2. VersionOperation을 inputQ에 추가
  3. Application에서 BTreeGetBulkOperation을 inputQ에 추가
  4. VersionOperation write
  5. VersionOperation 타임아웃 발생
  6. BTreeGetBulkOperation write
  7. 다음 handleIO에서 VersionOperation을 inputQ에 추가
  8. 서버는 BTreeGetBulkOperation을 먼저 수신함. 하지만 Version을 고려하지 않은 잘못된 Operation을 수신받았으므로 캐시 서버는 다운됨

Version에 따라 프로토콜이 달라지는 Operation의 경우, 즉.. https://github.com/naver/arcus-java-client/blob/master/src/main/java/net/spy/memcached/protocol/TCPMemcachedNodeImpl.java#L601-L628 코드의 플래그를 사용하는 Operation의 경우 클라이언트에서 Version이 완벽하게 수신하지 않았으면, 이 Operation에 대해서 inputQ에 추가하지 못하도록 cancel 처리해야 할 것 같습니다.

이 이슈에 대해 검토 바랍니다.

BulkSetWorker에서 정확한 실패 원인이 전달되게 수정.

BulkSetWorker의 awaitProcessResult() 메소드를 보면,
특정 set 연산이 실패할 경우에 END 응답을 전달하고 있다.
정확한 오류가 전달될 수 있게 수정되어야 한다.

    public void awaitProcessResult(int index) {
      try {
        boolean success = future.get(index).get(operationTimeout,
                TimeUnit.MILLISECONDS);
        if (!success) {
          errorList.put(
                  keys.get(index),
                  new CollectionOperationStatus(false, String
                          .valueOf(success), CollectionResponse.END));
        }
      } catch (Exception e) {
        future.get(index).cancel(true);
        errorList.put(keys.get(index), new CollectionOperationStatus(
                false, e.getMessage(), CollectionResponse.EXCEPTION));
      }
    }

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.