Friday, February 12, 2016

SQL Server Compact 4.0 SP1 hotfix available

I was recently made aware that a hotfix for SQL Server Compact was made available in 2014 –  but better late than never.

The hotfix can be requested via the Microsoft support knowledgebase here: https://support.microsoft.com/en-us/kb/2960153

The hotfix includes the following fixes:

2984052
FIX: Unexpected ordering when you run ORDER BY statement on a column that contains "-" in SQL Server Compact 4.0

2984053
FIX: "ConnectionString cannot be null" when you use folder redirection with offline files

2979868
FIX: It takes longer time than flush interval to flush committed transactions in SQL Server Compact 4.0

Sunday, January 24, 2016

SQL Server Compact & SQLite Toolbox 4.4 & 4.5 – Visual Guide of new features

After more than 470.000 downloads, version 4.5 of my SQL Server Compact & SQLite Toolbox extension for Visual Studio 2010 and later is now available for download and available via Tools/Extensions and Updates in Visual Studio. This blog post is a visual guide to the new features, improvements and bug fixes included in this release, many suggested by users of the tool via the GitHub issue tracker

I have moved the source code and issue tracker to GitHub, and I am planning to move the remaining CodePlex content “soon”. I have also moved the source of the scripting API that the Toolbox uses extensively to the same GitHub repository. As always, and suggestion, bug reports and pull requests are welcome!

New icons

The icons of the extension have all been overhauled, and updated to use the latest VS 2015 icons/images. From users already trying out the latest bits, I have received positive feedback on this visual change.

image

Query editor toolbar redesign and more

I have also started using Azure AppInsights to track (anonymously) feature usage etc. This revealed (maybe not surprisingly) that the most use feature is the query editor. So for this release, the query editor has received a lot of attention. In addtion to the changes below, the general layout of the editor has been streamlined (no more rounded corners).

image

The first button is new, and will open a “New Qery” window against the same database file. An options dropdown have been added to expose some of the (maybe overlooked) options that are available for the query editor. Setting these only affects the current query window, use the general settings (which you can reach via the Setting button in the query editor now) to change the “default” value for a setting.

Lets have a quick look at each of the options:

“Show result in Grid”: By default, the query results (from a SELECT statement) will be displayed as text for performance reasons, but you can opt-in to have the results displayed in an Excel like grid instead.

“Show Binary Values in Result”: By default, binary values (from for example “image” or “rowversion” columns) will be displayed a “System.Byte[]”. Enabling this will instead display the actual value as a hexadecimal string – but may affect performance.

“Show null values as NULL”: Until recently, NULL values were displayed as an empty string, not as “NULL”. Disabling this reverts to the old behaviour.

“Ignore DDL errors”: Enabling this (per query window only) feature, allows you to run the database creation scripts generated by the Entity Framework “Generate Database from Model” feature, and ignore errors caused by for example pre-existing tables.

“Use classic (plain) grid”: This will revert to using the “old” grid for grid results – about the “new” grid, see below!

New result grid in query editor

A new grid that allows you to further drill into the results of a query without writing any SQL is now the “default” result grid in the query editor. The grid uses http://wpfextendeddatagrid.codeplex.com/

image

“Autofilter” :The grid has built in filters, in the example above to filter on invoices for a particular customer (or even set of customers).

image

“Grouping”: Add columns you want to group by to the surface above the grid. In the example above, Invoices are grouped by Customer Id.

Options now part of Visual Studio options dialog

image

The options for the Toolbox have finally been integrated in the standard Visual Studio Tools/Options pages.

Improved SQL Server support

SQL Server Connections defined in Server Explorer now appear in the Toolbox, in order to better expose the SQL Server releated features of the Toolbox:

image

And scripting of a SQLite schema (only) based on a SQL Server database is now possible:

image

Improvements

A number of smaller improvements are also included in this release:

Improve format of INSERT statements generated by CSV import (to make the more readable)
Added latest System.Data.SQLite version (1.0.99)
Added link to blog post to DataAccess.cs to better explain how to use it.
Support for row-returning SQLite statements beginning with "WITH"
Added "Feedback" button
Using Azure Application Insights for metrics and crash reports
Ability to remove password from password protected SQL Compact file (by leaving password box blank)
Progress messages during export now displayed in Output window
Enable scripting of hierarchyid as varbinary(892)
Now able to script Adventureworks2014 schema and data to both SQL Compact and SQLite without errors
Table Data script (INSERTs) now have "RESEED" command at end always
Option to Opt out of Telemetry added

Bug fixes

In addition to the above improvements and features, and number of bug fixes are included in this release:

API:
Fix bug in SQLMetal.exe detection (on some configurations)
SQLite CREATE TABLE datetime default should be current_timestamp, not GETDATE()
Improved SQLite datetime handling in SELECT and Edit in Grid

User Interface:
CSV export from the SQL editor included one separator too many in the heading
Editor keyboard bugs fixed
Scroll issues in treeview fixed
Autohide of Toolbox bug fixed

Table builder:
Table builder was not scripting default values
Table builder should not script default value for IDENTITY columns

Thursday, August 20, 2015

SQL Server Compact & SQLite Toolbox 4.3 – Visual Guide of new features

After more than 415.000 downloads, version 4.3 of my SQL Server Compact & SQLite Toolbox extension for Visual Studio 2010 and later is now available for download and available via Tools/Extensions and Updates in Visual Studio. This blog post is a visual guide to the new features, improvements and bug fixes included in this release, many suggested by users of the tool via the CodePlex issue tracker

This new version was released in July at the time that Visual Studio 2015 went live, and is of tested with Visual Studio 2015 – keep in mind that all extensions now work with the free Visual Studio Community Edition. This release has a couple of new features that improves on the SQLite support originally added in version 4.0

Scan solution for SQLite database files

image

The “Add Connections from Solution” feature has been enhanced to scan all projects in the current solution for any SQL Server Compact and SQLite files. If any are found (based on the file patterns defined in Options), the Toolbox will attempt to add them as new connections.

The current file patterns are defined as shown below, but you can change them to anything you like:

image

Rename Connection name

The names of connections currently defaults to simply the file name of the database file. This becomes an issue, if you have many same named files in different folders connected. Therefore, you can now rename the Connection name:

image 

SQLite-net DataAccess.cs improvements

The SQLIite-net code generation feature has received a couple of small updates based on user feedback:

image

The generated classes will also contain any views defined in the SQLite database, and nullable values will be mapped accordingly (for example int?).

NULL value display

image

NULL values are now displayed as 'NULL' in Query results text and grid. Remember that you can see query results both as text and in a grid, the default is text for performance reasons, but this can easily be changed in Options:

image

Also notice that if you prefer the previous behaviour, this can be changed here, via the ShowNullValuesAsNULL option.

Other improvements

Unique indexes now always scripted as UNIQUE constraints (a long standing omission, which hit me when I was creating the reverse engineering for the SQL Server Compact providers for Entity Framework 7)

As usual, any feedback and comments are welcome, and if you like my free tools, I am always very grateful for a review on Visual Studio Gallery

Monday, July 27, 2015

Getting started with the SQL Server Compact and other providers for Entity Framework 7 beta 6

In connection with the recent release of Entity Framework 7 (EF7) beta 6 on NuGet, I have contributed two SQL Server Compact providers that work with this beta release (in close collaboration with and with great support from the EF7 team.)

Notice that the general “EntityFramework” package for EF7 is no longer available. Going forward you always install the desired provider packages as needed. For example EntityFramework.SqlServer

The SQL Server Compact providers

The SQL Server Compact providers are fully featured providers that support all required Core APIs (maybe with the exception of provider specific metadata). This includes support for returning IDENTITY values after INSERTs, Migrations, database object creation, and modelling via both the fluent API and Data Annotations. It also includes support the for Reverse-Engineer command, allowing you to create an EF7 model from an existing database.

For a walkthrough of how to get started with the SQL Server Compact providers, see the wiki article here. Initial Entity Framework 7 documentation is available here (a work in progress). As always feedback is very much appreciated, suggest you use the Github issue tracker for bug reports, feature suggestions or even Pull Request discussions and submissions!

Other providers

With the current EF7 beta, the following providers are available:

EntityFramework.SqlServer Provider for SQL Server/Azure SQL Database - supplied by Microsoft. This provider will be able to take advantage of the .NET Core support for System.Data.SqlClient on Unix and Mac. Currently only SQL Server 2012 and later is supported, but support for SQL Server 2008 is planned

EntityFramework.Sqlite Provider for SQLite, mainly for devices (tablet and phone) - supplied by Microsoft. Will also work with Xamarin (Andriod and IOs)

EntityFramework.InMemory In-memory provider for use with testing, supplied by Microsoft

All the providers above are hosted here on Github.

EntityFramework.Npgsql Provider for PostgressSQL - also works on Mono. Supplied by Shay Rojansky and others (expected to be released with beta7) – Github

EntityFramework.SqlServerCompact40 Provider for SQL Server Compact 4.0, supplied by yours truly – Github

EntityFramework.SqlServerCompact35 Provider for SQL Server Compact 3.5 SP2, also supplied by yours truly – Github

The current plan is to release EF7 at the same time as ASP.NET 5, which is currently scheduled (no guarantees) to beginning of next year.

Monday, July 13, 2015

SQL Server Compact ADO.NET data access performance – part 3: SELECTs

In the third instalment in this series, I will demonstrate how to implement SELECTs using the two available APIs in the SQL Server Compact ADO.NET provider. Keep in mind that the “alternative” method of SELECTing requires a corresponding index, so it will on work for simple scenarios where you are looking up on the primary key, for example – luckily, this is quite a common scenario.

I will implement the following interface method, which finds a cacheelement by it’s ID:

CacheElement FindElementByKey(Guid key);

Let’s first look at the implementation of the classic T-SQL SELECT based method:

public CacheElement FindElementByKey(Guid key)
{
    using (var command = _connection.CreateCommand())
    {
        command.Parameters.Add("@key", SqlDbType.UniqueIdentifier).Value = key;
        command.CommandText = @"SELECT [Tag], [Value], [CreatedAt], [ExpirationAt]
            FROM CacheElement
            WHERE [Key] = @key;";
        using (var reader = command.ExecuteReader())
        {
            if (reader.Read())
            {
                var result = new CacheElement();
                result.Key = key;
                result.Tag = reader.GetValue(1) == DBNull.Value ? null : reader.GetString(1);
                result.Value = (byte[])reader[1];
                result.CreatedAt = reader.GetDateTime(2);
                result.ExpirationAt = reader.GetDateTime(3);
                return result;
            }
            else
            {
                return null;
            }
        }
    }
}

This implementation uses a standard T-SQL based parameterized SELECT query. Nothing new here.

Now let us have a closer look at the implementation using TableDirect mode:

public CacheElement FindElementByKey(Guid key)
{
    using (var command = _connection.CreateCommand())
    {
        command.CommandType = CommandType.TableDirect;
        command.CommandText = "CacheElement";
        command.IndexName = "PK_CacheElement";

        using (var reader = command.ExecuteReader())
        {
            reader.Seek(DbSeekOptions.FirstEqual, key);
            if (reader.Read())
            {
                var element = new CacheElement();
                element.Key = key;
                element.Tag = reader.GetValue(1) == DBNull.Value ? null : reader.GetString(1);
                element.Value = (byte[])reader.GetValue(2);
                element.CreatedAt = reader.GetDateTime(3);
                element.ExpirationAt = reader.GetDateTime(4);
                return element;
            }
        }
        return null;
    }
}

Notice the required differences: The CommandType is set to TableDirect, the CommandText is set to the name of the table we want to query, and the IndexName property is set to the name of the index to be used. The Seek command will force a call to read to find the first matching column matching the key value. Notice that the key parameter is an params array, allowing you to use multi-column indexes.

In terms of performance, the classic SELECT appears to be 3 times slower than the TableDirect Seek, but of course your mileage may vary.

You can download the sample code from here.

Wednesday, April 29, 2015

SQL Server Compact ADO.NET data access performance – part 2: INSERTs

In this second part of this series, I will show by examples how to use the two available ADO.NET APIs in the SQL Server Compact ADO.NET provider for INSERTing data. I will also show some informal performance measurements, but keep in mind that your scenario may give different results.

In the sample solution, I will create a data access library for maintaining the following table, which you could be imagine could be used in a caching library:

CREATE TABLE CacheElement (
    [Key] uniqueidentifier NOT NULL,
    Tag NVARCHAR(4000) NULL,
    Value image NOT NULL,
    CreatedAt DATETIME NOT NULL,
    ExpirationAt DATETIME NOT NULL);

ALTER TABLE [CacheElement]
    ADD CONSTRAINT [PK_CacheElement] PRIMARY KEY ([Key]);

This table will represent the following class:

public class CacheElement
{
    public Guid Key { get; set; }
    public string Tag { get; set; }
    public Byte[] Value { get; set; }
    public DateTime CreatedAt { get; set; }
    public DateTime ExpirationAt { get; set; }
}

I will implement the following interface methods in the two possible ways, and then compare implementation and timings, by calling the library from a Unit test project:

    public interface IDataAccess
    {
        void InsertElement(CacheElement element);
        void InsertElements(List<CacheElement> elements);
    }

(I will add methods and implement further in the upcoming blog posts)

First let us look at the implementation of the T-SQL based INSERT, using best practices with explicit parameters (in the ClassicDal class) :

public void InsertElement(CacheElement element)
{
    using (var command = _connection.CreateCommand())
    {
        command.Parameters.Add("Key", SqlDbType.UniqueIdentifier).Value = element.Key;
        command.Parameters.Add("Value", SqlDbType.Image).Value = element.Value;
        command.Parameters.Add("CreatedAt", SqlDbType.DateTime).Value = element.CreatedAt;
        command.Parameters.Add("ExpirationAt", SqlDbType.DateTime).Value = element.ExpirationAt;
        if (String.IsNullOrWhiteSpace(element.Tag))
        {
            command.Parameters.Add("Tag", SqlDbType.NVarChar, 4000).Value = DBNull.Value;
        }
        else
        {
            command.Parameters.Add("Tag", SqlDbType.NVarChar, 4000).Value = element.Tag;
        }

        command.CommandText = @"INSERT INTO CacheElement
            ([Key],Tag,Value,CreatedAt,ExpirationAt)
            VALUES   
            (@Key, @Tag, @Value, @CreatedAt, @ExpirationAt)";
        command.ExecuteNonQuery();
    }
}

For all tests, I am using best practices for SQL Compact connection handling, and passing an already open connection. This avoids measuring the overhead of loading the SQL Compact engine DLL files, and opening the database file. Notice that for NULLable values, special handling has to be made.

Now lets us look at the implementation that uses the “raw” APIs for INSERTs (in the RawDAL class) :

public void InsertElement(CacheElement element)
{
    using (var command = _connection.CreateCommand())
    {
        command.CommandType = CommandType.TableDirect;
        command.CommandText = "CacheElement";
        using (var resultSet = command.ExecuteResultSet(ResultSetOptions.Updatable))
        {
            SqlCeUpdatableRecord record = resultSet.CreateRecord();
            record.SetGuid(0, element.Key);
            if (String.IsNullOrWhiteSpace(element.Tag))
            {
                record.SetValue(1, DBNull.Value);
            }
            else
            {
                record.SetString(1, element.Tag);
            }
            record.SetBytes(2, 0, element.Value, 0, element.Value.Length);
            record.SetDateTime(3, element.CreatedAt);
            record.SetDateTime(4, element.ExpirationAt);
            resultSet.Insert(record);
        }
    }
}

Notice the following special pattern: Setting the CommandType to TableDirect, the CommandText is the table name, we use the CreateRecord() method to get a SqlCeUpdateableRecord with “slots” that match our table columns. You have to know the exact “ordinal position” of your columns, you can get that by scripting a CREATE TABLE statement with my Toolbox, or inspecting the INFORMATION_SCHEMA.COLUMNS table. There are typed SetXxx methods that must match the datatype of your columns. Finally, call the Insert method to add the “record”.

In the sample code, I have also implemented methods to insert many rows in a single method invocation, thus saving the overhead of recreating a number of objects for each INSERT. This is similar to the way I insert data with my SqlCeBulkCopy library.

First the timings for a single INSERT: Raw: 31 ms, Classic: 29 ms – hardly any difference. Then 1000 single INSERTs, minimal object reuse: Raw: 2548 ms, Classic: 1936 – in this case not any advantage of the raw api. Then finally 1000 INSERTs, maximum object reuse: Raw: 73 ms, Classic: 354 ms. A significant difference, if that is a pattern you have in your application, ie inserting many rows in a single call.

You can download the code sample from here, and stay tune for the next installment: SELECT (using SetRange and Seek)

Monday, April 20, 2015

SQL Server Compact ADO.NET data access performance – part 1: Overview

In this five part series on SQL Server Compact performance, I will demonstrate and measure the performance of the two available data access APIs provider by the SQL Server Compact ADO.NET provider (System.Data.SqlServerCe.dll). The information in this blog post applies to both version 3.5 and 4.0. I will not cover usage of the unmanaged OLEDB interfaces (using C and C++)

As mentioned, the ADO.NET provider provides two APIs for data access (CRUD) operations:

One: The T-SQL standards based APIs - SqlCeCommand: ExecuteReader, ExecuteNonQuery, ExecuteScalar, that allows you to use text based SQL statements, including parameterized SQL. Most ORM implementations that support SQL Server Compact, like LINQ to SQL and Entity Framework, use this API, with the exception of the high perfromant OpenNETCF.ORM Framework, which is able to take advantage of

Two: The “table direct” APIs that allow you to bypass the T-SQL parsing and interpretation process, and that alone provides an obvious saving in processing. These include:

SqlCeResultSet (that inherits from SqlCeDataReader) - in particular when used with a SqlCeCommandType of TableDirect

SqlCeUpdateableRecord

SqlCeCommand.ExecuteResultSet

SqlCeDataReader.Seek

SqlCeCommand.SetRange

As you can see, none of these is available with the SQL Server ADO.NET objects, and they are particular to SQL Server Compact. This blog post series will demonstrate how to use these as alternatives to the “standard” methods.

The diagram from Laxmi’s blog post here explains the advantage graphically:

clip_image002

The “table direct” APIs bypass the Query Processor layer, and instead uses an ISAM like access pattern.

In the next blog post in this series, I will compare INSERT with SqlCeResultSet.Insert( SqlCeUpdateableRecord ) and demonstrate how to use the “Table Direct” APIs for INSERTs.