Emojis are not returned correctly by the adapter about tds

[ericlathrop]

I think I've got a similar issue with a fancy quote:

mssql-cli:

DB> select newstitle from hrdb_news where newsidentifier = 'Breeders_Cup_winner_Whitmore_sprints_to_8_year_old_campaign_123';
+--------------------------------------------------------------+
| newstitle                                                    |
|--------------------------------------------------------------|
| Breeders’ Cup winner Whitmore sprints to 8-year-old campaign |
+--------------------------------------------------------------+
(1 row affected)
Time: 0.355s
DB> select master.dbo.fn_varbintohexstr(cast(newstitle as varbinary(max))) from hrdb_news where newsidentifier = 'Breeders_Cup_winner_Whitmore_sprints_to_8_year_old_campaign_123';
-[ RECORD 1 ]-------------------------
(No column name) | 0x42007200650065006400650072007300192020004300750070002000770069006e006e0065007200200057006800690074006d006f0072006500200073007000720069006e0074007300200074006f00200038002d0079006500610072002d
(1 row affected)

where the column looks like:

+-----------------------+------------------+------------+----------+--------+---------+------------+----------------------+------------------------+------------------------------+
| Column_name           | Type             | Computed   | Length   | Prec   | Scale   | Nullable   | TrimTrailingBlanks   | FixedLenNullInSource   | Collation                    |
|-----------------------+------------------+------------+----------+--------+---------+------------+----------------------+------------------------+------------------------------|
| NewsTitle             | nvarchar         | no         | 510      |        |         | no         | (n/a)                | (n/a)                  | SQL_Latin1_General_CP1_CI_AS |

but Tds gives me:

iex(2)> IO.inspect(a.news_title, binaries: :as_strings)
"Breeders\x19  Cup winner Whitmore sprints to 8-year-old campaign"
<<66, 114, 101, 101, 100, 101, 114, 115, 25, 32, 32, 67, 117, 112, 32, 119, 105,
  110, 110, 101, 114, 32, 87, 104, 105, 116, 109, 111, 114, 101, 32, 115, 112,
  114, 105, 110, 116, 115, 32, 116, 111, 32, 56, 45, 121, 101, 97, 114, 45, 111,
  ...>>

with ecto schema:

schema "hrdb_News" do                           
  field(:news_title, :string)                   

[ericlathrop]

My issue was solved by setting up :tds_encoding per the instructions in the README.

[mjaric]

With erlang builtin :unicode module there is not higher compatibility between latin1 encoding and SQL_Latin1_General_CP1_CI_AS collation since some stuff is sent as ucs2 string. This ucs2 is NOT! UTF-16 nor UTF-8, ucs2 is fixed char code length (always 2 or 4 bytes long) while UTF-X is variable in char code length. If you need to store anything that has chars codes that contain higher byte other than zero you should use :tds_encoding lib