Dan Maharry

Writing about web development since 1997

Simple.Data.Docs and Sample Into 2013

On March 27 last year, Mark Rendle put an idle call out into the ether for help with the documentation for his Simple.Data project. At a loose end, I said yes...

It has been a busy nine months in the world of Simple.Data.Docs. Mark has been busy, releasing no fewer than 16 new builds of Simple.Data.Core from v0.16 to v0.18.3 / 1.0.0-rc3 since that time which have been collectively downloaded some 9110 times according to nuget.org. I'd like to hope that the documentation that's being slowly assembled is to your satisfaction. It's gratifying to note that since adding in some counters in early October, we've had some 22600 page views (14500 unique) from 4500 visits (1700 unique) to the doc site and 2 complaints so it can't be all bad.

A few thanks then to

  • Marcus Hammarberg for his coverage of testing with the InMemoryAdapter
  • Jeff Schumacher for his Advanced Naming Scenarios commit
  • Code52 for their metro.css project. The new Metro-style docs have certainly made the site a nicer place to visit and work on.

I'm about 70% finished on the major, basic topics (based on the Simple.Data.SqlServer provider) and, assuming that I don't have to pause efforts for another three months to write a book, the core of the docs should be complete for the first year anniversary. Hurrah!

It's important to note that I've also been updating the Simple.Data.Sample project to map to and expand on those examples I've included in the docs. Although it hasn't been updated on NuGet since I took over, you'll find the code itself completely refreshed with 100+ new query samples to look at in addition to those in Mark's own test suites. Once the core docs are finished and Simple.Data v1 is finally released, hopefully Mark will be kind enough to update the Sample package on NuGet as well. I'd also like to release it as a VB package and as a set of nUnit tests as well once that has happened. Currently it is a C# command line application.

Once the core docs are complete, I'll add in updates and extra pages for helper functions in the API as they are requested or queried about on the group, but there are a couple of areas I'd like to look at

  • Incorporating the docs for other Simple.Data providers into the site. Vagif Abilov has completed his Simple.OData docs and we'll work together on how best to integrate the two. Hopefully, the same can happen with other provider owners.
  • Documenting the Simple.Data adapter and provider models and how to write your own.
  • Real-world scenarios and gotchas.

It's this last area in particular, that I'd like to ask your assistance with. Whether you're querying MongoDB, SQLAnywhere, oData, SQL Server, Azure, or something else, if you've come across and then resolved any sticky issues using Simple.Data which you'd be happy to share with the world, please get in touch or leave a comment below. It would be great to hear from you.

Inferred Hierarchies and Evaluation Strategies in Simple.Data

This is the second of a number of posts derived from the documentation for Simple.Data that I’m compiling and writing at the moment. The code here is part of the Simple.Data.Sample project on github.

Simple.Data is bloody clever at times but quite easy to get confused by. The most common confusion is between using commands that return multiple results as a SimpleQuery object and those that return a single results as a SimpleRecord object. With the return type hidden under the veil of ‘dynamic’ it’s easy to lose track and try to call a method on one that only works on the other. It’s the main reason for Mark to deprecate the FindBy method for v1.0. The second most common is around the subject of lazy\eager-loaded JOINs and what that means in your code.

More...

NotMetro: A Theme For dotNetBlogEngine

I uploaded my first theme for dotNetBlogEngine today. You’ll find it at http://dnbegallery.org/cms/List/Themes/NotMetro. It’s powering the very blog you’re reading now. It’s called NotMetro as it borrows from the official Metro styles document but not too much. And of course, it’s not called Metro any more, is it.

Here are the specs.

Layout is one wide, 900px column. The widget zone is in the footer at the end of the page. All standard widgets look fine in it.

Any and all feedback would be gratefully received

DDD Southwest Session Notes 4 : Redis Is the New Black

The final session of my day at DDD Southwest was an introductory talk by Chris Hay about Redis, “an open-source advanced key-value store”. It introduced Redis, looked at some of the basic features and commands and their possible applications. There’s full documentation of all Redis commands here.

So What Is Redis then?

Released on April 10 2009 Salvatore Sanfilippo (@antirez)
Open source, sponsored by VMWare

It is an open source, in-memory key value data store, which does more than just keys and values. (so NoSQL with knobs on.) It also does sets, lists & counters. It was written in C by Salvatore Sanfilippo (@antirez) and is very very fast. Comparisons by use as a cache server would include AppFabric& memcached. It is used by stackoverflow, reddit, & others.

Redis runs on Linux. There is a WIndows build-ish, but use Linux.

Redis is a server – you use a client or a client API library connect to it. A command-line client is installed wit the server, but there are many thrid party redis clients – listed here. For .NET developers, there are two good options.

Booksleeve uses a lot of clever async stuff with Task<T> but the demos all used ServiceStack.Redis because it is easier to work with and simpler to read.

Working with KEYS

There are three redis commands to work with key value pairs – SET, GET & EXPIRE

redis-cli SET MyKey "Hello World"
redis-cli GET MyKey
redis-cli EXPIRE MyKey 10  <- after 10 seconds

In C#

using ServiceStack.Redis;

using (var cli = new RedisClient(IPaddress))
{ 
  ViewData[MyKey] = cli.Get<string>("MyKey"); 
  // cli.Set<string>("MyKey", "Hello World");
  // cli.Expire("MyKey",10);
}

Redis Keys are useful for Caching and Sessions. There is a Microsoft session provider written for use against redis. (SQL Server provider really sucks)

Working with COUNTERS
Commands: increment, decrement, increment by, decrement by

Default increment is 1

redis-cli incr  aCounter 
redis-cli decr  aCounter 
redis-cli incrby aCounter 2 
redis-cli decrby aCounter 2

Redis runs on a single thread so it is completely thread-safe. incr, decr ain't going to produce clashes and same results for same things. Counters useful for web page hit counters, perf counters, analytics, unique ids, sequences etc.

Working with SETS
Commands: sadd, smembers, srem

SADD MySet C   (adds C to MySet) 
SMEMBERS MySet (gets the members of MySet) 
SREM MySet C   (removes C from MySet)

Remember that sets are unordered and all members are unique. The power of sets comes in the set arithmetic

SUNION MySet1 MySet2 
SDIFF MySet1 MySet2
SINTER MySet1 MySet2

You can find the rest of the commands here.

Q\A : Redis deals with strings typically, but it is generally pretty good with other types as it stores things at byte level. JSON Serialization works well but prefer to avoid it if possible.

Sets are useful for tagging (a la Stack Overflow).

SORTED SETS

Useful for indexes, object graph relationships.

Demo of ChukGraph
(or How I almost got myself shot in the head by gangsters...)

LISTS

(Queue - first in first out)
LPUSH mylist A
RPOP mylist

(Stack - last in last out)
RPUSH mylist A
LPOP mylist

PIPELINING

Typically, you send command to Redis without waiting for a response. A reply is returned in response to each request. However, it is possible to pipeline/queue requests and wait for a response which is a cumulative result of all the requests.

TRANSACTIONS

Redis sort of supports transactions

Atomic : yes
Consistent : need to add a WATCH command for that
Isolated : Yup - be aware of threads though (single threaded only)
Durability : Kind of, but the more durable you want it (persisted to disk for instance), the slower redis is. It takes about 10ms to do this. (Which is longer than you want - really)

Redis also supports LUA Scripting

PUB\SUB

PUBLISH channel message
SUBSCRIBE, UNSUBSCRIBE channel  

REPLICATION

You can set up a group of Redis servers in a master-slave configuration

  • Slaves are read-only
  • Why do this? Redis is single-threaded, so push intensive queries to the slaves to prevent blocking and have them publish the changes back to master

At some point in the future, there will be Redis clustering.

http://redis.io

DDD Southwest Session Notes 3 : Defensive Programming 101

The third session of my day at DDD Southwest was a talk by Niall Merriganentitled Defensive Programming 101. Or, as it turned out, “Top 10 Things You Shouldn’t Forget To Do To Start Securing Your Website”. This was probably my favourite of the sessions I attended at DDD Southwest. Niall’s a funny bloke with useful things to teach so if you get a chance to see him talk, do.

Further information and resources can be found on Niall’s site here at http://www.certsandprogs.com/2012/05/dddsw-roundup-and-resources.html. You can contact him on Twitter at @nmerrigan

Writing secure code is hard and takes time. Sales people do not care and we trust each other  that our code is secure. Not that many people try to screw up a site they are visiting. We write for general users.

The Top 10 "Screwables"

10. Restrict your HTTP Headers. It can tell those who care too much, such as the webserver being used, .net version etc. Don’t forget also to restrict easy access to sensitive files. Kill elmah.axd, trace.axd. Search for the phrase googlehacks and you’ll find the number of ways google can help the hacker attack your site simply because you’ve allowed them to index your *.axd files, *.pst files etc.

9. Passwords. Don't use the same password for every single site. Cross-pollination hurts. Don’t save passwords in plain text connection strings. Encrypt them, change them regularly and don’t ell them to others. NB You can't encrypt your password in a connection string when using the entity framework.

8. Patching. Try and follow the server patch bulletins sent out by Microsoft and make sure you test your web applications with new patches attached as they can break.

7. Validation. Don't rely only on client-side scripting for validation. Turn off javascript in your browser and make sure server-side validation is working. Prefer whitelists to blacklists. Validate to RFC rules. Use a central validation source – i.e. a single place to update validation rules.

Search for e.g. RFC email - shoud point to ISO standard validation routine

6. Email & Custom Errors. Email presents too much information to users, and not enough to developers. Error messages should only show users what they expect to see. No stacktraces etc. Very ditto for emails. Turn CustomErrors ON IN PRODUCTION with pages handling specific error codes – 404, 500 etc. Use web config transforms to kill it or set it to localonly and set up custom errors. Handle your errors correctly.

5. Database and AppPool Permissions. Don't use sa in your connection string. Don’t create a user and set it to dbo. Use two connection strings, one for reading, one for writing. Never run AppPools as *Admin user accounts. Always use the minimum permissions. Don't give access to SQL tables if sprocs and views are all that’s required. Perhaps give user only execute permissions if you only use sprocs. (Yes this should include insert perms. Oh, and don’t use dynamic sql in sprocs)

4. Directory traversal. How do I send a file to client that opens a save as dialog? Ans: Whatever you do, don’t do something like this: download.aspx?file.txt. Because download.aspx?..\web.config will work. IIS isn't handling and therefore automatically blocking the request at this point, your handler is. You could probably download the whole source code in dll files and decompile it. If you used the asp.net website template, you can get all the cs files as well. In short, don't use a file name for downloads. Use a GUID or include a hash for the file to check you're downloading the file you think it is.

Find link on blog to WAM_USER WAM_PASSWORD metabase vulnerability on Niall's site,

3. Injection attacks. Addition of scripts to your site against your will. Don't set enablePageValidation to false. Very silly. Don't use cookieless sessions. Only make cookie on server-side readonly.

2. SQL Injection. nuff said. Don't allow naked SQL in your code.

1. Users will never do what you think they are going to do.